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1.0  Introduction 


Id  1990,  the  U.S.  Air  Force  initiated  a  phase  in  the  Military  Construction 
(MTT  ,rON)  Process  known  as  Project  Definition.  The  intention  of  the  Air  Force  in 
initiating  this  phase  was  twofold:  first,  to  inq)rove  customer  satisfaction  with  the 
facility  provided,  and  second,  to  reduce  design  changes  throughout  the  construction 
project.  Specifically,  Project  Definition  is  that  phase  in  the  mihtary  construction 
process  in  which  an  architect-engineering  firm  or  an  in-house  designer  collects  user 
requirements  and  then  conducts  on-site,  preliminary  analysis  and  design. 

1.1  Purpose 

The  puipose  of  this  research  is  to  investigate  the  effectiveness  of  the  Project 
Definition  phase  in  achieving  its  intended  purposes.  The  hypothesis  is  that  if  the 
Project  Management  Team  for  an  Air  Force  construction  project  fixUy  and  effectively 
follows  the  Project  Definition  process,  as  outlined  in  The  U.S.  Air  Force  Project 
Manager ’s  Guide  to  Project  Definition,  then  a  more  successful  design  effort  will 
result. 

1.2  Scope 

This  research  will  focus  only  on  the  Project  Definition  phase  of  the  MELCON 
process.  To  evaluate  the  effectiveness  of  Project  Definition,  twenty-five  Air  Force 
Center  for  Environmental  Excellence/Construction  Management  Office 
(AFCEE/CMO)  projects  were  studied.  The  projects  were  each  chosen  because 
Project  Definition  and  design  had  been  conq)leted.  The  projects  are  of  varying  types 
and  sizes,  ranging  from  an  aircraft  parking  apron  to  a  dormitory  renovation  and 
ranging  in  cost  from  $1.0  Million  to  $19.5  Milhon.  While  this  research  was  conducted 
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primarily  for  AFCEE,  its  results  should  benefit  other  Major  Commands  and 
construction  project  managers  throughout  the  Air  Force. 

1.3  Thesis  Organization 

Chapter  2  of  this  thesis  consists  of  a  backgroimd  explanation  of  the  MILCON 
process.  Project  Definition,  and  AFCEE  organization.  Also,  the  Construction  Industry 
Institute’s  research  into  pre-project  planning  as  well  as  metrics  and  benchmarking  are 
briefly  discussed  in  Chapter  2.  Chapter  3  describes  the  methodology  used  for  this 
research,  including  an  e?q)lanation  of  the  matrices  used  to  evaluate  the  projects,  the 
development  of  the  research  questionnaires,  and  a  fisting  of  the  projects  studied. 
Chapter  4  presents  the  results  of  the  research  and  an  analysis  of  the  results.  Chapter  5 
presents  conclusions,  and  Chapter  6  discusses  recommendations. 
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2.0  Bacl^ound 


2.1  The  U.S.  Air  Force  Military  Construction  Process 

The  process  by  v\4iich  an  Air  Force  construction  project  is  conceived,  funded, 
designed  and  conq>leted  is  complex,  lengthy,  and  heavily  regulated.  It  involves 
numerous  people  from  several  levels  of  the  U.S.  government.  Figure  2. 1  shows  a 
rough  timeline  of  the  Mihtary  Construction  (MILCON)  process  from  the  initial  request 
for  the  project  to  the  start  of  the  construction  phase  (USAF  CE  1991).  The  following 
is  a  brief  explanation  of  the  Air  Force  MILCON  process;  it  is  not  intended  to  be 
exhaustive.  For  a  more  in-depth  discussion  of  the  process.  The  United  States  Air 
Force  Project  Manager ’s  Guide  for  Design  and  Construction  ( 199 1 )  is 


recommended,  from  which  this  explanation  was  taken. 


Figure  2.1 

USAF  Military  Construction  Process 
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Typically,  to  initiate  the  MILCON  process,  an  organization  on  an  Air  Force 
base  requests  a  new  facility,  an  addition,  or  renovation  of  an  existing  fecility  through 
the  base  civil  engineering  (BCE)  squadron.  If  the  request  is  vahd  and  fits  within  the 
base’s  long-range  plans,  the  BCE  is  responsible  for  conq)leting  a  Department  of 
Defense  Form  1391,  vMch  outlines  the  project  scope,  cost,  and  features.  This  form  is 
sent  with  other  requests  for  construction  fi'om  the  base  to  a  Major  Command 
(MAJCOM),  which  oversees  the  operations  of  several  different  bases.  Afl:er  the 
constmction  requirement  is  approved  at  this  level,  each  MAJCOM  then  sends  all  of 
the  requests  for  construction  to  Headquarters  U.S.  Air  Force/Civil  Engineer  (HQ 
USAF/CE).  HQ  USAF/CE  compiles,  validates,  and  approves  or  disapproves  all 
requests  sent  fi^om  the  MAJCOMs.  If  the  project  is  approved,  HQ  USAF/CE  issues  a 
Planning  Instruction  (PI)  to  the  MAJCOM.  The  PI  is  the  formal  notification  that 
advanced  planning  on  the  project  may  begin  at  the  command’s  discretion. 

The  PI  authorizes  the  MAJCOM  to  initiate  design  activities,  and  generally,  at 
this  time,  the  Requirements  and  Management  Plan  (RAMP)  is  coitq)leted.  The  RAMP 
consists  of  two  parts:  the  requirements  part  consisting  of  a  brief  project  description, 
general  and  special  requirements,  and  a  parametric  cost  estuuate^  and  a  project 
management  plan  including  a  Hsting  of  the  project  management  team  and  project 
strategic  decisions. 

Also  at  this  time,  a  Design  Agent  is  appointed  by  the  government.  Due  to  U.S. 
congresaonal  regulations,  the  design  and  construction  phases  of  most  Air  Force 
projects  are  managed  by  the  U.S.  Army  Corps  of  Engineers  or  the  U.S.  Navy  Facilities 
Engineering  Command  (NAVFAC).  Thus,  the  Design  Agent  is  uhimately  responsible 
for  the  design  of  the  project  and  all  government  dealings  with  private  firms. 

The  next  phase  in  the  process  is  the  selection  of  the  Architect-Engineering  firm 
(A-E)  who  whl  design  the  project.  Usually,  interested  A-Es  respond  to 
aimouncements  made  in  the  Commerce  Business  Daily,  which  carries  listings  of 
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project  desciptions,  qualifications  required,  and  critical  need  dates.  The  A-E  is 
selected  based  on  recent  specific  ej^erience  and  e>^ertise  related  to  the  project.  For 
current  Air  Force  projects,  the  A-E  usually  must  demonstrate  past  experience  with 
conducting  on-site  data  gathering,  analysis,  and  design  charrettes.  Charrettes  will  be 
defined  and  explained  in  Section  2.2.  Occasionally,  the  Deaga  Agent  may  decide  to 
have  the  design  conducted  by  in-house  government  personnel  rather  than  a  private 
A-E  firm 

Once  the  A-E  is  selected,  project  participants  meet  at  the  Pre-Definition 
Conference.  This  conference  is  attended  by  the  requesting  base  organization  (facility 
end  user  or  future  occupant),  the  BCE,  the  A-E,  the  Air  Force  Project  Manager,  and 
the  Design  Agent.  At  the  conference,  general  design  requirements  are  estabhshed, 
base  design  and  environmental  standards  are  reviewed,  all  project  team  member 
responsibilities  are  defined,  and  a  cost  for  services  is  requested  fi^om  the  A-E. 

The  pre-definition  conference  begins  the  Project  Defimtion  phase,  \\^iich  will 
be  discussed  in  detail  in  Section  2.2.  At  the  end  of  Project  Definition,  a  preliminary 
design  and  a  parametric  cost  estimate  (PCE)  are  produced  by  the  A-E.  Rather  than  a 
detailed  cost  estimate  based  on  quantities  of  various  materials,  the  parametric  cost 
estimate  is  an  estimate  based  on  facility  type,  square  footage,  and  the  estimator’s 
experience.  The  PCE  developed  during  Project  Definition  is  used  to  update  the  DD 
Form  1391,  which  is  then  sent  to  the  MAJCOM  and  subsequently  to  HQ  USAF  for 
budget  preparation.  The  Office  of  the  Secretary  of  Defense  then  reviews  each 
service’s  construction  requirements  and  prepares  the  final  Department  of  Defense 
budget.  The  defense  budget  is  then  included  in  the  President’s  budget  and  sent  to 
Congress.  After  Congress  reviews  the  defense  budget,  it  is  approved  through  an 
authorization  bill  and  funded  by  means  of  an  appropriations  biU.  Each  MILCON 
project  is  a  line  item  in  the  appropriations  bill.  If  the  President  signs  the  bill,  each  of 
the  included  projects  is  fimded  for  five  years. 
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Once  funding  for  the  project  is  received,  the  construction  contract  can  be 
awarded.  When  the  awarded  contractor  has  been  given  a  Notice  to  Proceed, 
construction  of  the  facility  may  begin.  Typically,  this  is  about  four  years  after  the 
initial  request  for  the  facility  was  made. 

2.2  Project  Definition 

Project  Definition  was  initiated  by  the  Air  Force  in  1990  to  improve  the  facility 
end  user’s  satisfaction  with  the  product  provided  and  to  reduce  changes  throughout 
the  construction  project.  Prior  to  this,  the  requesting  organization  was  sometimes  not 
involved  in  the  planning  process,  and  occasionally  there  was  little  interaction  between 
the  facility  user  and  the  facility  designer.  This  resulted  in  numerous  design  and 
construction  changes,  meaning  extensive  time  delays,  huge  cost  increases,  and  a 
&cility  user  or  base  commander  that  was  not  always  satisfied  with  the  final  product. 

Realizing  that  the  best  time  to  gather  design  inputs,  discuss  alternatives,  and 
make  changes  was  at  the  beginning  of  the  project,  the  Air  Force  made  the  decision  to 
change  the  MILCON  process.  Figure  2.2  illustrates  the  relationship  between  project 
costs  and  the  ability  to  influence  the  project  over  time  (CII  Pub.’s  39-1  and  39-2). 

The  Project  Definition  phase  was  added  to  provide  a  forum  in  which  all  project  team 
members,  including  the  facility  user,  the  decision  maker,  and  facility  designer,  could 
readily  exchange  design  information  at  the  earliest  point  possible  in  the  project.  The 
following  paragraphs  offer  a  brief  ejqplanation  of  Project  Definition  as  summarized 
fi'om  The  U.S.  Air  Force  Project  Manager ’s  Guide  to  Project  Definition  (1994). 
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PROJECT  TIME 


NOTE:  Alphanumeric-Project  time  period  In  which  the  design  consists  only  of  written 
descriptions  and  quanitities. 

Drawing— Project  time  period  in  which  the  design  descriptions  are  translated 
into  working  drawings. 

Construction-Project  time  period  in  which  the  facility  is  constructed. 

Occupancy— Time  period  In  which  the  facility  user  has  begun  to  use  the  facility 
for  Its  intended  purpose.  _ 


Figure  2.2 

Influence  and  Expenditures  Curve  for  the  Project  Life  Cycle 

As  shown  in  Figure  2.1,  Project  Definition  begins  at  the  predefinition 
conference  with  the  selected  Architect-Engineer  (A-E)  or  in-house  designer  and  ends 
wlien  the  A-E  has  presented  a  preliminary  design  and  a  parametric  cost  estimate.  It 
normally  involves  two  phases:  a  requirements  analysis  phase  and  a  schematic  design 
phase.  Project  Definition  typically  lasts  four  to  twelve  weeks,  depending  on  the 
p.ompleYity  of  the  project.  A  normal  Project  Defimtion  schedule,  lasting  ei^t  weeks, 
is  shown  in  Figure  2.3  (USAF  CE  1994). 

To  adequately  bring  all  design  requirements  into  consideration.  Project 
Definition  must  involve  many  different  people  with  differing  interests  in  the  project. 
The  core  group  of  individuals  that  must  execute  Project  Definition  are  known  as  the 
Project  Management  Team  (PMT).  The  PMT  usually  includes  the  Air  Force  Project 
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Monday  Tuesday 

Friday  j 

VUeekI 

A-E  distributes  questionnaire.  | 

Users  complete  questionnaire. 

A-E  researches  building  type. 

VVh«k2 

A-E  compile 
conducts 

s  data  from  questionnaire  and 
>  interviews  and  site  visits. 

Weeks 

A-E  conducts  CHARRETTE  A-E  conducts 

on-srte  analysis  ^  ^ 

Week4 

Weeks 

A-E  produces  Project  Definition  documents  v 

Including  parametric  cost  estimate. 

Weeks 

Week? 

Project  Team  reviews  Project  Definition  Package. 

weeks 

A-E  finalizes  Project  Definition  Package. 

A-E  presents 
to  HRPWC. 

Figure  2.3 

Normal  Project  Definition  Schedule 


Manager,  the  Users’  Representative,  the  Design  Agent,  the  base  Contracting  Officer, 
the  Base  Civil  Engmeeiing  Representative,  and  the  A-E.  Other  organizations  on  the 
base  which  also  participate  in  Project  Definition  in  varying  degrees  are  known  as  the 
Base  Support  Team,  and  they  normally  include  the  Security  Police,  the 
Communications  Squadron,  the  base  Fire  Department,  Environmental  Management, 
the  Safety  Office,  and  Base  Operations.  Organizations  not  directly  associated  vsdth  the 
base,  known  as  Outside  Agencies,  may  also  influence  the  process  and  may  include  the 
local  or  state  historical/cultural  agencies,  the  state  or  federal  Environmental  Protection 
Agency,  utility  conqianies,  and  other  regulatory  agencies.  Finally,  the  project  decision 
maker,  known  as  the  Highest  Ranking  Person  With  Concern  (HRPWC),  must  also  be 
involved  in  the  process  for  it  to  be  successful. 

The  portion  of  the  Project  Definition  phase  in  vsliich  the  A-E,  facflity  user,  and 
the  rest  of  the  PMT  meet  and  discuss  the  project  requirements  is  known  as  the 
charrette.  The  purpose  of  the  charrette  is  to  fully  develop  and  quantify  the  functional 
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and  technical  requirements  of  the  project.  For  normal-sized  projects,  the  charrette 
lasts  one  week;  for  conqtlex  projects,  two  sq)arate  charrettes  for  analysis  and  design 
may  be  required.  Although  the  A-E  leads  the  charrettes,  the  Air  Force  Project 
Manager  must  ensure  the  charrette  stays  on  schedule  and  that  all  necessary  personnel 
are  involved  in  the  process. 

Project  Definition  consists  of  two  phases.  The  first  of  these  phases  is  the 
requirements  analysis  phase,  also  known  as  programming  in  most  architectural  design 
schedules.  In  this  phase,  the  A-E  distributes  questionnaires,  makes  visits,  conducts 
research  and  interviews,  and  performs  any  necessary  site,  geotechmcal,  or 
environmental  investigations.  Prior  to  the  start  of  the  charrette,  the  A-E  produces 
summaries^  graphic  displays,  and  question  sheets  so  space  and  requirements  issues 
come  to  the  fi'ont  during  the  charrette  meetings.  The  requirements  analysis  charrette 
produces  products  which  are  alphanumeric  in  nature:  goal  statements,  tabulated  space 
projections,  analysis  cards  of  project  issues,  site  analysis  cards,  environmental 
concerns,  affinity  matrices,  area  summaries,  flow  charts,  bubble  diagrams,  value 
engineering  opportunities,  cost  estimates,  and  site  photographs.  The  desire  of  the  Air 
Force  is  for  the  PMT  to  reach  a  consensus  regarding  the  project  requirements  during 
the  analysis  portion  of  the  charrette.  As  shown  in  Figure  2.2,  it  is  relatively  easy  and 
inexpensive  to  change  requirements  while  the  project  is  still  in  the  alphanumeric  stage. 

The  second  phase  of  Project  Definition  is  the  schematic  design  phase.  Once 
agreement  has  been  reached  concerning  the  project  requirements,  the  A-E,  with  the 
participation  of  the  rest  of  the  PMT,  begins  to  create  a  schematic  solution.  The  design 
charrette  involves  e?^loring  options  and  design  alternatives,  and  eventually  reaching  a 
consensus  on  a  schematic  design.  The  detail  to  wiiich  the  project  is  designed  is  not 
strictly  governed.  Some  building  systems  such  as  the  floor  plan  may  be  designed  to 
50%,  while  others  such  as  the  electrical  system  may  only  be  designed  to  5%.  It  is 
critical  that  all  members  of  the  PMT  be  present  during  this  phase  and  that  the  HRPWC 
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is  briefed  regularly  on  the  layout  of  the  facility.  Also  critical  is  the  presence  of  the  cost 
estimator.  It  is  the  estimator’s  re^onsibility  to  provide  cost  estimates  for  aU  design 
alternatives  as  they  are  being  discussed  and  to  make  suggestions  on  how  to  lower  the 
inififll  and  operating  costs;  this  is  to  keep  the  project  from  exceeding  its  programmed 
budget.  Products  of  the  design  charrette  normally  include  a  site  development  plan,  a 
schematic  floor  plan,  furniture  and  equipment  layouts,  building  area  tabulations, 
exterior  building  elevation  drawings,  base  coiiq)rehensive  plan  conformance,  building 
sections,  building  subsystem  cost  analysis/value  engineering,  design  criteria  and 
selected  building  systems  narrative,  the  parametric  cost  estimate,  an  operabiUty  and 
maintainability  report,  and  an  environmental  checkhst. 

Following  the  charrettes,  the  A-E  conqjiles  all  design  decisions  into  a  Project 
Definition  Package.  This  package  is  presented  in  a  notebook  format  and  includes  such 
things  as  a  project  description,  site  development  and  floor  plans,  narratives, 
environmental  concerns,  and  the  cost  estimate.  The  final  act  of  the  A-E  diiring  Project 
Definition  is  to  brief  the  HRPWC  on  the  preliminary  design  and  parametric  cost 
estimate.  By  doing  so,  the  project’s  design  is  “cast  in  concrete”  and  later  changes  are 
reduced. 

2.3  The  U.S.  Air  Force  Center  for  Environmental  Excellence 

The  research  for  this  thesis  was  done  for  and  in  cooperation  with  the  U.S.  Air 
Force  Center  for  Environmental  Excellence  (AFCEE),  located  at  Brooks  Air  Force 
Base,  Texas.  AFCEE  was  created  in  1991  to  ensure  environmental  and  facihty 
excellence  for  the  U.S.  Air  Force  (AFCEE  1994).  While  the  majority  of  the  AFCEE 
functions  involve  Air  Force  environmental  concerns,  they  also  include  a  branch  solely 
devoted  to  constraction  management.  AFCEE’ s  Construction  Management  Office 
manages  constmction  projects  for  several  of  the  Air  Force’s  smaller  Major 
Commands.  As  of  the  date  of  this  thesis,  the  construction  projects  AFCEE  manages 
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constitute  approximately  40%  of  the  total  number  of  Air  Force  housing  construction 
projects,  8%  of  the  Base  Reahgnment  and  Closure  projects,  and  20%  of  regular 
military  construction.  The  other  Air  Force  construction  projects  are  managed  by  the 
larger  commands,  including  Air  Combat  Command,  Air  Mobility  Command,  Air 
Education  and  Trainmg  Command,  and  Air  Force  Materiel  Command.  While  this 
research  only  examines  twenty-five  projects  managed  by  AFCEE,  tiie  results  should 
also  apply  to  the  other  Major  Commands  as  well  since  Project  Definition  is  an  Air 
Force- wide  initiative. 

2.4  The  Construction  Industry  Institute’s  Pre-Project  Planning  Research 

The  Construction  Industry  Institute  (CII),  located  in  Austin,  Texas  is  a 
consortium  made  up  of  approximately  ninety  owners  and  construction  contractors  and 
includes  private  firms  as  well  as  government  organizations.  The  stated  goal  of  Cn  is; 
to  improve  the  total  quality  and  cost  effectiveness  of  the  construction  industry, 
thereby  providing  a  competitive  advantage  for  U.S.  business  in  the  global 
marketplace  (Cn  1991).  CII  accomplishes  its  goal  by  fimding  construction  research 
through  the  University  of  Texas  at  Austin  and  other  universities  throughout  the  United 
States.  One  of  CD’s  major  research  efforts  was  the  study  of  the  effort  and  subsequent 
payoff  of  up-front  project  planning,  known  as  Pre-Project  Planning  (P^).  This 
research  was  extensive  and  resulted  in  the  development  of  CII  Publication  39-2,  the 
Pre-Project  Planning  Handbook. 

The  cn  research  team  defined  pre-project  planning  as  the  process  of 
developing  sufficient  strategic  information  with  which  owners  can  address  risk  and 
decide  to  commit  resources  to  maximize  the  chance  for  a  successful  project  (Cn  Pub 
39-2).  Through  its  research,  Cn  defined  the  four  major  fimctions  and  sub-fimctions  of 
pre-project  planning  to  be  the  foUovsdng: 
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1.  Organize  for  Pre-Project  Planning 

a.  Select  Team 

b.  Draft  Charter 

c.  Prepare  Pre-Project  Planning  Plan 

2.  Select  Project  Altemative(s) 

a.  Analyze  Technology 

b.  Evaluate  Site(s) 

c.  Prepare  Conceptual  Scopes  and  Estimates 

d.  Evaluate  Alternatives 

3.  Develop  a  Project  Definition  Package 

a.  Analyze  Project  Risks 

b.  Document  Project  Scope  and  Design 

c.  Define  Project  Execution  Approach 

d.  Estabhsh  Project  Control  Guidelines 

e.  Compile  Project  Definition  Package 

4.  Decide  Whether  to  Proceed  with  Project 

a.  Make  Decision 

To  vaUdate  the  need  for  pre-project  planning,  CD  conducted  a  study  of  sixty- 
two  projects  and  compared  the  level  of  pre-project  planning  effort  and  project  success 
for  each  project.  The  study  produced  conclusive  evidence  that  a  definite  correlation 
between  pre-project  planning  effort  and  the  level  of  project  success  does  exist;  the 
greater  the  pre-project  planning  effort,  the  greater  the  success  of  the  project.  The 
research  team  also  calculated  a  conservative  estimate  of  20%  savings  versus 
authorization  estimate  by  consistently  performing  pre-project  planning  versus 
performing  very  little  pre-project  planning  (Cn  Pub.  39-1). 

In  the  development  of  this  thesis,  the  author  conq)ared  the  CII Model  with 
the  new  Air  Force  MILCON  process.  It  is  the  opinion  of  the  author  that  the  new 
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MILCON  process  contains  every  element  of  the  Cn  Model.  The  Air  Force’s 
Requirements  and  Management  Plan  satisfies  the  first  of  the  Cn  pre-project  planning 
fimctions  by  naming  the  project  management  team  and  stating  project  strategic 
decisions.  During  Project  Definition,  the  Air  Force  evaluates  and  selects  alternatives, 
and  the  A-E  prepares  the  project  definition  package.  Finally,  when  Project  Defimtion 
is  conq)lete  and  a  preliminary  design  and  cost  estimate  are  achieved,  the  Air  Force, 
Department  of  Defense,  and  U.S.  Congress,  with  information  produced  during  Project 
Definition,  decide  whether  or  not  to  proceed  with  the  project.  If  executed  effectively, 
the  new  Air  Force  MILCON  process  should  produce  savings  conq)arable  to  those  of 
the  cn  companies. 

2.5  The  Construction  Industry  Institute’s  Metrics  and  Benchmarking  Research 

cn  has  defined  the  term  metric  as  being  a  quantifiable,  sinq)le  and 
understandable  measure  which  can  be  used  to  optimize  performance.  The  term 
benchmarking  is  defined  as  a  continuous  process  of  inqjrovement  or  the  search  for 
industry  practices  that  lead  to  superior  performance  (Tucker  1995).  In  other  words, 
metrics  are  indicators  of  performance,  while  benchmarking  is  the  process  by  which 
performance  is  improved  The  U.S.  Air  Force  has  defined  the  eight  attributes  of  a 
good  metric  as  the  following  (AFSC  1990); 

1.  It  is  accepted  as  meaiungful  to  the  customer. 

2.  It  tells  how  well  an  organization’s  processes  and  tasks  fulfill  its  goals  and 
objectives. 

3.  It  is  single,  understandable,  logical,  and  repeatable. 

4.  It  shows  a  trend. 

5.  It  is  unambiguously  defined. 

6.  Its  data  is  economical  to  collect. 

7.  It  is  timely. 

8.  It  drives  the  “appropriate  action.” 

The  cn  Benchmarking  and  Metrics  Committee  is  currently  atten:q)ting  to 
define  metrics  that  can  be  apphed  to  the  entire  construction  industry.  To  organize  the 
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many  metrics  relatiBg  to  the  construction  industry,  the  committee  created  a  tier 
system,  as  shown  in  Figure  2.4  (Cn  B&M  1995).  The  research  conducted  for  this 
thesis  included  the  development  of  metrics  for  the  Air  Force’s  Project  Definition 
Phase.  In  comparison  to  the  CD  Metrics  Tiers,  the  Project  Defimtion  metrics  would 
be  considered  Tier  HI  metrics  since  this  phase  is  only  one  step  within  the  Air  Force’s 
project  planning  system 


Figure  2.4 
cn  Metrics  Tiers 


2.6  Other  Background  Sources 

Several  other  sources  were  reviewed  by  the  author  in  preparation  of  this  thesis. 
The  fill!  titles  of  the  following  sources  are  listed  in  the  Bibliography.  EVr.  James 
Broaddus’s  dissertation  (1991)  contains  a  study  of  fifty-five  U.S.  Navy  construction 
projects  and  compared  the  level  of  up-front  planning  and  subsequent  project  success. 
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As  with  the  Cn  Pre-Project  Planning  study,  the  Broaddus  study  shows  conclusively 
that  a  direct,  positive  correlation  exists  between  the  level  of  pre-project  planning  and 
project  success.  Aniello  Tortura’s  thesis  ( 1993)  and  Dr.  Michele  Hamilton’s 
dissertation  (1994)  each  support  different  aspects  of  the  Cn  Pre-Project  Planning 
research  previously  described.  Finally,  Steven  Davis’s  thesis  (1993)  includes  a  study 
of  twenty-two  Air  Force  projects,  and  like  the  Cn  and  Broaddus  studies,  it  shows  the 
correlation  between  planning  effort  and  project  success. 
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3.0  Methodology 


3.1  Metrics 

As  discussed  in  the  previous  chapter,  to  study  the  effectiveness  of  Project 
Definition,  metrics  were  developed  to  give  an  indication  of  performance.  The  metrics 
vdiich  measure  the  outcome  of  the  design  effort  are  generally  related  to  the  following: 
the  cost  estimate  vs.  the  actual  cost,  the  cost  of  the  Project  Defimtion  and  design 
efforts,  the  impact  of  design  changes  on  the  design  schedule,  and  customer 
satisfaction.  However,  to  fiiUy  evaluate  Project  Definition,  not  only  were  the  outputs 
of  the  design  phase  measured,  the  inputs  were  measured  as  weD.  The  measurement  of 
each  of  these  metrics  will  be  discussed  in  the  following  sections. 

3.2  Objectives  Matrix 

To  stay  within  the  limits  of  this  thesis,  it  was  necessary  to  produce  a  single 
Project  Definition  input  and  Design  outcome  rating  for  each  project.  In  order  to 
measure  the  performance  of  several  different  metrics  and  still  have  only  one  input  and 
output  rating  per  project,  an  objectives  matrix  was  used  to  produce  a  numerical 
performance  index.  The  objectives  matrix  allows  the  flexibility  to  combine  a  variety  of 
criteria,  weights,  measuring  systems  and  performance  ratings  into  a  single  performance 
index  (Cn  Pub.  8-1).  Specifically,  this  method  allows  both  subjective  measures  (such 
as  customer  satisfaction)  and  objective  measures  (such  as  estimate  vs.  actual)  to  be 
combined  numerically.  The  objectives  matrix  works  by  scoring  each  criterion  on  a 
scale  of  1  to  10  and  then  multiplying  this  score  by  the  criterion’s  assigned  weight, 
producing  a  numerical  value  for  each  criterion.  After  each  criterion’s  value  is 
calculated,  all  of  the  values  are  added  to  produce  the  index  for  one  project. 
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3.2.1  Development  of  Matrices 

The  hypothesis  of  this  thesis  is  that  if  the  Project  Management  Team  effectively 
follows  the  Project  Definition  process,  a  more  successful  design  effort  will  result.  By 
measuring  both  inputs  to  and  outputs  of  the  process,  a  Project  Definition  Index  and  a 
Design  Success  Index  for  each  project  was  produced.  Assigning  these  indices  (rated 
fi-om  0  to  1000)  as  axes  on  a  graph  produces  a  single  point  for  each  project  (see 
Figure  3. 1).  Plotting  all  twenty-five  projects  on  the  same  graph  should  provide  an 
indication  of  whether  Project  Definition  is  achieving  its  desired  purposes.  Ideally,  a 
diagonal  line  fi-om  the  0,0  point  to  the  1000,1000  point  will  result,  indicating  that 
lower  rated  inputs  into  the  design  process  produce  lower  output  ratings  and  higher 
inputs  produce  higher  outputs.  This  means  that  those  projects  to  which  the  Project 
Definition  process  is  eflfectively  applied  will  have  more  successful  design  efforts. 
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Figure  3.1 

Expected  Results  Graph 


Management  personnel  at  the  AFCEE  Construction  Management  Office  and 
the  author  met  several  times  over  a  two  month  period  to  discuss  the  development  of 
the  input  and  output  matrices  used  in  this  study.  For  measurement  of  the  inputs  to 
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Project  Definition,  based  on  the  United  States  Air  Force  Project  Manager’s  Guide  to 
Project  Definition  the  author  suggested  several  factors  to  the  AFCEE  managers. 

These  factors  primarily  dealt  with  the  people  involved  in  the  process,  the  products  of 
the  process,  as  well  as  the  process  itself.  The  AFCEE  managers  refined  the  hst  of 
input  criteria,  and  the  Project  Definition  Matrix  was  formed.  Since  the  list  of  input 
criteria  is  extensive,  the  criteria  were  broken  down  into  three  sub-matrices:  the 
Project  Management  Team,  the  Project  Definition  Process,  and  the  Project  Definition 
Products. 

For  the  output  matrix  (Design  Success  Matrix),  several  possible  metrics  were 
discussed  for  inclusion  in  the  study.  The  primary  source  of  measurement  information 
came  from  the  AFCEE  managers’  experience  with  mihtary  construction.  The  AFCEE 
persoimel  were  primarily  concerned  about  design  changes,  the  cost  of  these  changes, 
and  the  in:q)act  changes  had  on  the  project  schedule.  They  were  also  concerned  about 
the  level  of  customer  satisfaction  with  the  new  MILCON  process.  These  metrics  were 
refined  and  formed  the  basis  of  the  Design  Success  Matrix  and  Customer  Satisfaction 
Sub-Matrix. 

The  criteria  on  the  matrices  were  weighted  by  the  deputy  director  of  the 
AFCEE  Construction  Management  OfBce  and  the  chief  of  the  Reserve  Division  of  the 
AFCEE  Construction  Management  Office.  As  the  matrices  were  revised  during  the 
two  months  of  meetings  and  discussion,  the  weights  assigned  to  the  criteria  were  also 
revised. 

3.2.2  Organization  of  Matrices 

The  general  organization  of  the  matrices  is  shown  in  Figure  3.2.  The  Project 
Definition  Matrix  and  its  sub-matrices  are  shown  in  Figures  3.3  to  3.6.  The  Design 
Success  Matrix  and  Customer  Satisfaction  Sub-Matrix  are  shown  in  Figures  3.7 
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and  3.8.  The  Definitions  of  the  different  criteria  for  each  matrix  are  provided  in  the 
figures. 

On  the  Design  Success  Matrix,  one  criterion,  the  “Cost  of  Design”,  had  two 
different  scoring  scales.  For  projects  ranging  fi’om  $1. 0-5.0  Million,  the  median  rating 
(5)  for  “Cost  of  Design”  was  9.5%  of  the  total  budget  (as  diown  in  Figure  3.7),  wMe 
projects  with  a  total  budget  greater  than  $5.0  Million  were  scored  with  the  median 
rating  being  6.5%.  This  was  done  to  ensure  that  all  of  the  projects  were  treated  on  an 
equal  basis.  The  two  median  values  (9.5%  and  6.5%)  were  used  since  these  were  the 
Corps  of  Engineers  accepted  standards  for  Cost  of  Design  for  these  sizes  of  projects. 


Figure  3.2 

Organization  of  Matrices 
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Performance  Criteria 
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Note  1:  Includes  A-E,  Facility  User,  BCE,  HRPWC, 
and  Design  Agent. 


Rgure  3,3 

Project  Definition  Matrix 
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INDEX 

0 

Definitions: 

A-E:  Architect-Engineer  firm  or  in-house  design  team  chosen  to  execute  Project  Definition 
Cost  Estimator:  Professional  cost  estimator  employed  by  A-E  to  provide  the 
parametric  cost  estimate  (PCE)  for  the  project. 

Design  Agent:  Corps  of  Engineers,  NAVFAC,  or  Air  Force 

Facility  User:  USAF  Facility  End  User/Future  Occupant 

HRPWC/Decision  Maker:  USAF  Highest  Ranking  Person  With  Concern 

BCE:  USAF  Base  Civil  Engineering  staff 

MAJCOM:  The  major  command  responsible  for  the  project. 

Base  Support  Any  other  on-base  organizations  v^h  input  into  the  project. 

Outside  Agencies:  Non-military  organizations  vMh  input  into  the  project  (i.e.  EPA,  Utilities). 
Project  Manager:  USAF  Project  Manager 


Figure  3.4 

Project  Management  Team  Sub-Matrix 
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Definitions: 

User  Req'ts  Defined;  how  well  user's  requirements  defined  during  PD. 

Project  Goals  Defined:  how  well  project  goals  identified  before  design  started. 

PD  Process  Explained:  how  well  the  PD  process  was  communicated. 

Design  Alternatives  Considered;  the  degree  to  which  design  alternatives 
were  considered. 

Consensus  Reached:  how  well  the  team  agreed  on  the  selected  design 
alternatives. 

Right  People  Involved:  the  degree  to  which  the  right  people  involved  in  the 
process. 

Charrette;  how  well  the  team  conferences  produced  a  satisfied  customer, 
lower  cost,  and  reduced  schedule. 

Library  Wall:  how  effective  the  temporary  wall  display  of  information  was 
in  communicating  design  requirements  and  decisions. 

Note  1:  Includes  A-E,  BCE,  Facility  User,  HRPWC  and  Design  Agent. 


Figure  3.5 

Project  Definition  Process  Sub-Matrix 


Definitions: 

Project  Management  Plan:  communicates  strategic  decisions  and  defines  project  team. 

Pre-Definition  Conf:  conference  with  A-E  that  reviews  scope,  requirements,  standards,  etc. 

Data  Gathering/Req'ts  Analysis:  portion  of  PD  that  produces  design  requirements 
information. 

Site  Plan:  drawing  that  depicts  site  requirements,  including  utilities,  contours,  and  parking. 

Floor  Plan:  drawing  that  depicts  facility  interior  space  requirements. 

Building  Sub-Systems:  narratives  that  describe  sub-systems  such  as  walls,  HVAC, 
comm,  etc. 

Elevations:  drawings  that  depict  exterior  appearance. 

Cost  Estimate:  the  parametric  cost  estimate  developed  during  PD. 

Environmental  Checklist:  list  of  critical  environmental  issues  that  the  team  should 
consider. 


Figure  3.6 

Project  Definition  Products  Sub-Matrix 
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Performance  Criteria 
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100%  Estimate  vs.  PCE:  The  percentage  of  change  between  the  cost  estimate 
developed  during  Project  Definition  and  the  estimate  at  Design  Complete. 

Bid  Cost  vs.  PCE:  The  percentage  of  change  between  the  parametric  cost  estimate 
developed  during  Project  Definition  and  the  bid  cost  of  the  project. 

100%  Estimate  vs.  Programmed  amount;  The  percentage  of  change  between  the 
amount  originally  budgeted  and  the  cost  estimate  at  Design  Complete. 

Cost  of  Changes  During  Design:  Percent  of  budget  spent  on  Air  Force-directed 
changes. 

Cost  of  Project  Definition  Effort:  Percent  of  budget  spent  on  the  Project  Definition 
phase. 

Cost  of  Design:  Percent  of  budget  spent  on  the  Design  phase  (inducing  PD). 

Impact  of  Changes  on  Schedule:  The  percentage  of  total  design  time  that  was 
spent  on  Air  Force-directed  design  changes. 

Note  1:  Includes  Fadllty  User,  BCE,  and  HRPWC  (see  Customer  Sat.  sub-matrix) 


Figure  3.7 

Design  Success  Matrix 
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Performance  Criteria 
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Definitions: 

Satisfied  with  Process:  Rating  of  satisfaction  with  PD  process. 

HRPWC  Involvement:  Decision  maker’s  satisfaction  with  his/her  involvement. 

Meets  Mission  Needs  of  Organization:  Degree  to  which  facility  will  meet 
the  mission  needs  of  the  using  organization  and  the  base. 

Functionality:  The  design  of  the  facility  \mII  enhance  the  job  effectiveness 
of  the  users. 

Maintainability  Compared  wth  other  facilities  of  this  type,  this  facility 
will  be  easy  to  maintain. 

Aesthetics:  The  facility  will  fit  in  well  with  the  architectural  scheme  of  the  base. 

Note1:  Includes  all  survey  interviev^es. 

Note  2:  Includes  Facility  Erid  User  and  HRPWC. 


Figure  3.8 

Customer  Satisfaction  Sub-Matrix 


3.3  Development  of  Questionnaires 

To  get  direct  input  for  the  matrices  developed  for  this  research,  written 
questionnaires  were  created  with  the  guidance  of  AFCEE  personnel.  Each  question 
on  the  survey  is  a  statement  about  one  particular  aspect  of  the  project.  The 
interviewee  was  asked  to  circle  a  rating  from  1  to  10,  indicating  his  or  her  agreement 
or  disagreement  with  each  statement.  In  some  cases,  additional  comments  were 
sought  for  certain  questions,  and  all  interviewees  were  asked  to  provide  comments  or 
suggestions  they  had  concerning  the  Project  Definition  process.  Also,  the  Air  Force 
Project  Managers  were  asked  to  fill  in  blanks  on  their  questionnaires  so  that  project 
execution  and  cost  data  could  be  recorded.  The  questionnaires  used  for  this  research 
are  fisted  as  Appendices  1  through  6.  Additionally,  the  conq)leted  matrices  for  one 
example  project  are  fisted  as  Appendix  7. 

3.4  Data  Sources 

As  shown  on  the  matrices,  the  data  for  each  criteria  came  from  a  variety  of 
project  sources.  This  was  done  so  that  the  results  of  the  study  would  not  be  biased 
toward  any  particular  project  participant  group.  Specifically,  the  customer  groups 
provided  data  on  the  Project  Definition  Process  and  Customer  Satisfaction,  while  the 
Air  Force  Project  Manager  provided  data  on  the  Project  Management  Team,  the 
Project  Defimition  Products,  project  costs  and  schedule.  AFCEE  considers  not  only 
the  facility  user  but  also  other  project  engineering  personnel  and  the  project  decision 
maker  to  be  its  customers.  The  customer  group  contained  both  privately  en^loyed 
and  government  employed  personnel,  and  the  government  persoimel  surveyed  included 
both  military  and  civil  service  personnel.  The  following  is  a  summary  description  of 
the  sources  that  were  utilized  to  gather  project  performance  data: 

Air  Force  Project  Manager:  for  this  study,  this  is  the  person  from  AFCEE  who 
manages  the  Project  Definition  and  Design  phases  for  the  Air  Force. 
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Architect-F.n  gineer:  this  is  the  project  designer  and  may  be  a  private  A-E  firm 
or  a  team  of  in-house  government  personnel  The  A-E  is  required  to  not  only  design 
the  facility  but  also  to  execute  the  Project  Definition  phase,  including  data  gathering, 
data  analysis,  charrette  execution,  and  cost  estimation. 

Base  Civil  F.n^neering  Staff  Representative:  this  is  the  person  in  the  local  Air 
Force  base  civil  engineering  squadron  vs^io  is  responsible  for  coordination  of  the 
project  at  his  or  her  base. 

Design  Agent:  usually  the  U.S.  Army  Corps  of  Engineers  or  U.S.  Naval 
Facilities  Engineering  Command  (NAVFAC);  ultimately  responsible  for  the  design  of 
the  project. 

Facility  User:  the  facility  end  user  or  future  occupant. 

Ffighest  Ranking  Person  With  Concern  fHRPWCl:  the  project’s  decision 
maker;  could  be  either  senior  ranking  military  or  civil  service  personnel. 

3.5  Projects  Studied 

The  projects  chosen  for  inclusion  in  this  study  are  listed  in  Table  3.1.  Project 
Definition  on  each  of  these  projects  was  conqjlete,  and  the  construction  contract  had 
been  awarded,  however  the  construction  was  not  yet  conq)lete.  This  was  done  so  that 
the  information  collected  about  Project  Definition  was  easily  remembered  by  the 
project  participants.  This  also  allowed  for  an  in-depth  study  into  the  effects  Project 
Defimition  has  on  the  design  of  the  project,  including  the  affect  design  changes  had  on 
the  design  schedule  and  the  variance  between  cost  estimates  and  actual  costs. 
However,  due  to  the  fact  that  construction  was  not  complete,  project  construction 
data  could  not  be  collected.  This  prevented  comparing  such  things  as  the  original  cost 
estimate  to  the  final  construction  cost  and  prevented  the  study  of  the  effects  Project 
Definition  has  on  the  final  outcome  of  the  project. 
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Flight  Simulator  Dobbins  AFB,  GA  6,000  Design-bid-build  Corps 

RED  HO^SE  Stmctural  Utility  Facility  Kelly  AFB,  TX  2,300  Design-bid-biald  Corps 

F-16Beddown  New  Orleans  NAS,  LA  8,200  Design-bid-build  NAVFAC 

Housing _ Barksdale  AFB,  LA _ 16,820  Design-bid-build  Ait  Force 


4.0  Presentation  and  Analysis  of  Results 


4.1  Questionnaire  Response 

Originally^  thirty-one  projects  were  chosen  for  inclusion  in  this  study,  with  186 
questionnaires  sent  to  project  participants  (six  participants  per  project).  Of  the 
questionnaires  distributed,  146  were  returned.  However,  due  to  lack  of  information  or 
non-conformity  with  the  purpose  of  this  thesis,  six  projects  were  eliminated  from  the 
study.  This  left  twenty-five  projects  that  were  included  in  the  final  tabulation  of  results 
and  analysis. 

4.2  Project  Definition  Ratings  versus  Design  Success  Ratings 

As  stated  in  Chapters  1  and  3,  this  study  hypothesized  that  those  projects  to 
which  the  Project  Definition  process  was  effectively  apphed  would  have  more 
successful  design  efforts  than  those  projects  to  which  the  process  was  not  effectively 
apphed.  When  the  questionnaire  responses  for  this  study  were  compiled,  the  matrices 
were  used  to  produce  ratings,  or  indices,  as  shown  in  Chapter  3.  This  produced  a 
Project  Definition  Index  and  a  Design  Success  Index  for  all  twenty-five  projects. 

Microsoft  Excel  software  was  used  to  tabulate  and  plot  the  data  in  graphic 
format.  Excel  produced  a  line  of  best  fit,  calculating  the  equation  of  this  line  using 
linear  regression.  The  value  was  also  calculated  by  the  Excel  software,  indicating 
the  rehabihty  of  the  line  of  best  fit.  The  scatter  plot  of  the  Project  Definition  and 
Design  Success  indices  and  the  line  of  best  fit  are  shown  in  Figure  4. 1. 

The  plot  shows  that  a  positive  correlation  exists  between  Project  Definition 
effort  and  Design  Success  effort.  This  indicates  that,  on  average,  higher  Project 
Definition  Indices  produce  higher  Design  Success  Indices.  In  practical  terms,  this 
means  that  when  the  entire  Project  Management  Team  meets  on-site  to  discuss  project 
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objectives,  goals,  and  requirements,  and  works  to  produce  a  preliminary  design,  cost 
estimate,  and  written  narratives,  then  a  design  phase  with  less  cost  variance  (between 
estimated  and  actual),  less  design  changes,  less  design  cost,  less  schedule  variance,  and 
greater  customer  satisfaction  wUl  result. 


Figure  4.1 

Project  Definition  v.  Design  Success 

As  can  be  seen  in  Figure  4. 1,  the  line  of  best  fit  has  a  slope  approximately 
equal  to  0.84  and  intercepts  the  y-axis  at  slightly  less  than  zero.  This  indicates  that  an 
approximate  one-to-one  relationship  exists  between  Project  Definition  effort  and 
Deagn  Success.  The  =  0.5158  indicates  that  the  line  of  best  fit  is  a  reliable 
representation  of  the  relationship  between  the  independent  variable,  Project  Definition, 
and  the  dependent  variable,  Design  Success.  Typically,  values  greater  than  0.5 
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indicate  a  strong  correktion  exists  between  the  independent  and  dependent  varkbles 
(Ingram  1977).  However,  it  should  be  noted  that  this  strong  correlation  does  not 
statistically  prove  a  cause  and  effect  relationship.  Also  apparent  on  the  graph  shown 
in  Figure  4. 1  is  the  “right-sidedness”  of  the  plotted  points,  indicating  that  in  all  but  one 
of  the  projects  studied,  the  Project  Definition  Indices  were  above  500.  Since  the  score 
of  “5”  was  considered  average  on  the  matrices,  this  indicates  that  these  projects  had 
above  average  ratings  for  the  Project  Management  Team,  Project  Defimtion  Products, 
and  Project  Definition  Process.  The  Design  Success  ratings  were  more  varied,  ranging 
fi'om  a  low  of  238  to  a  high  of  848. 

After  the  six  indices  (see  Figures  3.2-3. 8)  for  all  twenty-five  projects  were 
calculated,  average  values  were  calculated.  The  average  Project  Defimtion  Index  was 
725,  with  the  average  Project  Management  Team  Index  being  691,  the  average 
Products  Index  being  724,  and  the  average  Process  Index  being  762.  The  average 
Design  Success  Index  for  all  projects  was  577,  with  the  average  Customer  Satisfaction 
Index  being  812. 

4.3  Comparison  of  Projects  With  Charrettes  and  Projects  Without  Charrettes 

As  described  in  Chapter  2,  the  charrette  is  the  one  or  two-week  long  meeting 
vvhere  the  Project  Management  Team  gathers  to  discuss  reqxiirements  and  make  design 
decisions.  It  is  the  central  focus  of  the  new  MILCON  process  and  is  expected  to 
improve  teamwork  as  well  as  overall  design  success.  For  this  study,  of  the  twenty-five 
projects  studied,  fifteen  of  the  projects  had  charrettes  during  the  Project  Definition 
phase  and  ten  did  not  have  charrettes.  An  in-depth  analysis  of  the  data  provides 
evidence  that  on  those  projects  where  the  participants  met  at  a  charrette,  more 
accurate  and  concqjlete  preliminary  designs  and  estimates  were  gained.  Figure  4.2  is  a 
summary  of  the  con^arison  of  those  projects  that  had  charrettes  and  those  that  did 
not.  The  Design  Success  Metrics  are  the  same  criteria  that  were  used  in  the  Design 
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Success  Matrix  (see  Figure  3.7).  It  is  notable  that  the  projects  having  charrettes  out¬ 
performed,  in  every  success  metric,  those  that  did  not. 


Mean  Values  (%) 

P(T<‘t) 

two  tail 

All** 

Charrette 

No  Charrette 

100%Estv.  PCE: 

12.40 

8.07 

18.90 

Bid  Costv.  PCE: 

15.79 

13.29 

19.30 

0.4608 

100%  Extv.  PA: 

9.32 

7.13 

12.60 

0.3140 

0.76 

0.60 

1.00 

0.5113 

Cost  of  PD: 

2.30 

2.07 

2.61 

0.2986 

Cost  of  Design(<$5M)*: 

11.41 

11.38 

11.44 

0.9822 

Cost  of  Design  {>$5M)*: 

8.75 

7.86 

15.00 

0.0620 

Impact  of  Changes: 

6.25 

5.00 

8.0C 

0.5000 

Customer  Satisfaction: 

812 

827 

791 

0.4121 

NOTE:  *  Includes  cost  of  PD  (1 7  projects  <$5M,  8  projects  >$5M). 

**  15  projects  had  charrettes,  10  projects  did  not  have  charrettes. 


Definitions: 

100%  Estimate  vs.  PCE:  The  percentage  of  change  between  the  cost  estimate 

developed  during  Project  Definition  and  the  estimate  at  Design  Complete. 

Bid  Cost  vs.  PCE:  The  percentage  of  change  between  the  parametric  cost  estimate 
developed  during  Project  Definition  and  the  bid  cost  of  the  project. 

100%  Estimate  vs.  Programmed  amount:  The  percentage  of  change  between  the 
amount  originally  budgeted  and  the  cost  estinrrate  at  Design  Complete. 

Cost  of  Changes  During  Design:  Percent  of  budget  spent  on  Air  Force-directed 
changes. 

Cost  of  Project  Definition  Effort:  Percent  of  budget  spent  on  the  Project  Definition 
phase. 

Cost  of  Design:  Percent  of  budget  spent  on  the  Design  phase  (including  PD). 

impact  of  Changes  on  Schedule:  The  percentage  of  total  design  time  that  was 
spent  on  Air  Force-directed  design  changes. 

Customer  Satisfiietion:  Index  from  sub-matrix;  rated  from  0  to  1000. 


Figure  4.2 

Performance  Comparison  of  Projects  With  Charrettes  and 
Projects  Without  Charrettes 


The  last  column  in  Figure  4.2  shows  the  results  of  a  f  test  for  the  charrette  and 
non-charrette  means  diown.  The  null  hypothesis  was  that  the  sanqile  means  for  the 
charrette  and  non-charrette  proejcts  are  equal  using  a  significance  of  0. 10  due  to  the 
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small  sample  sizes.  The  last  column  gives  the  probability  that  the  sait^le  means  are 
equal  for  the  each  criterion.  The  results  of  the  t  test  indicate  that  the  null  hypothesis 
can  only  be  rejected  for  the  100%  Estimate  v.  Parametric  Cost  Estimate  and  the  Cost 
of Design  for  Projects  Greater  than  $5  Million  criteria.  This  indicates  that  the  rest  of 
the  criteria  are  statistically  similar  and  that  the  conq)arison  of  the  means  can  not  be 
taken  as  statistically  conclusive  evidence.  However,  by  inspection  of  the  entire  table  in 
Figure  4.2,  it  can  generally  be  concluded  that  the  projects  that  had  charrettes 
performed  better  than  those  projects  that  did  not  have  charrettes. 

The  sanqile  variances  for  the  Design  Success  Metrics  for  the  projects  with 
charrettes  and  projects  without  charrettes  are  shown  in  Figure  4.3.  Inspection  of  this 
figure  shows  that,  for  the  most  part,  those  projects  that  had  charrettes  appeared  to 
perform  much  more  consistently  than  the  projects  that  did  not  have  charrettes. 


Sample  Variances 

Design  Success  Metrics 

Charrette 

No  Charrette 

100%  Est  V.  PCE: 

61.78 

479.66 

Bid  Cost  v.  PCE: 

181.76 

652.68 

100%  Ext  V.  PA: 

92.27 

288.93 

Cost  of  Changes: 

1.40 

3.33 

Cost  of  PD: 

0.88 

2.37 

Cost  of  Design(<$5M)*: 

14.27 

61.53 

Cost  of  Design  (>$5M)*: 

18.81 

0.00 

Impact  of  Changes: 

134.62 

78.44 

Customer  Satisfaction: 

10240.21 

12125.56 

F^[ure  4.3 

Comparison  of  Variances 


Conq)aring  the  matrix  performance  ratings  of  projects  with  charrettes  and 
those  without  charrettes  also  shows  that  charrettes  produce  higher  ratings.  Figures 
4.4  and  4.5  highli^t  this  fact. 
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Figure  4.4 

Project  Definition  v.  Design  Success 
(Charrette  Projects  Bighlighted) 


PMT  Index 

Products  Index 

Process  Index 

Oust  Sat  Index 

A\«-all 

691 

724 

762 

812 

Ave-charrette 

743 

609 

724 

743 

765 

827 

Ave-no  charrette 

698 

530 

642 

696 

759 

791 

Figure  4,5 

Sununary  of  Average  Indices 


As  shown  in  Figures  4.4  and  4.5,  the  average  values  of  the  matrix  indices  are 
higher  for  those  projects  with  charrettes.  Particularly  noteworthy  is  the  difference 
between  the  average  values  for  the  Design  Success  Index  (609  v.  530).  Additionally, 
the  three  highest  non-charrette  projects  all  had  significant  on-site  site  data  gathering 
and  on-site  design  analysis,  although  they  did  not  have  charrettes.  If  these  three 
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projects  are  taken  out  of  the  calculation,  the  average  Design  Success  Index  for 
projects  not  having  charrettes  drops  from  530  to  441. 

4.4  Analysis  of  Selected  Survey  Questions 

Although  all  questions  asked  on  the  research  questionnaires  were  used  in  the 
calculation  of  the  matrix  indices,  a  few  of  the  questions  have  been  selected  for  further 
analysis.  These  questions  were  chosen  due  to  the  relative  weight  on  the  matrices 
assigned  to  them  by  AFCEE.  The  following  are  the  questions  and  the  average 
req)onses  to  them: 


Question:  The  A-E firm/in-house  design  team  chosen  to  conduct  the  charrettes 
during  the  Project  Definition  phase  of  this  project  was  well  prepared  and  effectively 
led  the  on-site  analysis  and  design  required  by  the  contract. 

Average  Rating  by  USAF  Project  Manager:  6.82  (SUghtly  Agree) 


Question:  As  the  Air  Force  Project  Manager,  I  feel  I  was  effective  in  bringing  the 
Project  Management  Team  to  consensus. 

Average  Rating  by  USAF  Project  Manager:  7.42  (Agree) 

Question:  The  Schematic  Floor  Plan  developed  during  Project  Definition  satisfied 
the  user  by  adequately  depicting  all  space  requirements  of  the  facility. 

Average  Rating  by  USAF  Project  Manager:  8.00  (Agree) 

Question:  The  facility  user ’s  requirements  were  clearly  defined  during  Project 
Definition. 

Average  Rating  by  A-E,  BCE,  Design  Agent,  User,  &  HRPWC:  8.08  (Agree) 

Question:  A  consensus  was  reached  among  project  team  members  on  selected  design 
alternatives  and  building  sub-systems. 

Average  Rating  by  A-E,  BCE,  Design  Agent,  &  User:  7.81  (Agree) 

Question:  I  was  satisfied  with  my  involvement  in  the  final  design  decisions  made 
during  Project  Definition. 

Average  Rating  by  HRPWC;  8.58  (Agree) 
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Question:  This  facility  will  enhance  the  mission  effectiveness  of  the 
base/organization. 

Average  Rating  by  User  &  BDRPWC;  8.88  (Strongly  Agree — ^Agree) 

Question:  This  facility  will  enhance  the  job  effectiveness  of  the  personnel  in  my 
organization. 

Average  Rating  by  User:  8.39  (Agree) 

Question:  /  was  satisfied  with  the  Project  Definition  process,  because  1  believe  the 
process  enhanced  the  final  product. 

Average  Rating  by  All  Project  Participants:  7.66  (Agree) 

This  question  was  also  averaged  by  participant  group: 

Average  Rating  by  A-E:  8.32  (Agree) 

Average  Rating  by  BCE:  7.53  (Agree) 

Average  Rating  by  Design  Agent:  7.83  (Agree) 

Average  Rating  by  User:  7.05  (Agree —  Slightly  Agree) 

Average  Rating  by  HRPWC:  8.74  (Strongly  Agree — ^Agree) 

Average  Rating  by  USAF  Project  Manager:  7.08  (Agree — Slightly  Agree) 


4.5  Participants*  Comments  and  Suggestions 

On  each  questionnaire,  project  participants  were  asked  to  provide  comments 
and/or  suggestions  they  had  on  inq)roving  the  Project  Definition  process.  These 
comments  are  listed  in  Appendices  8  through  13.  Many  of  the  survey  respondents 
provided  comments  pertaining  to  a  specific  project,  but  only  those  comments  directed 
toward  the  process  itself  are  included  in  the  appendices.  Several  general  observations 
can  be  made  by  reviewing  the  comments  and  suggestions  provided  by  the  project 
participants.  The  following  paragraphs  describe  these  observations. 

4.5.1  Satisfaction  with  Project  Definition/Charrette  Process 

As  shown  in  the  average  Customer  Satis&ction  Index,  the  Project 
Definition/Charrette  process  is  greatly  favored  by  most  project  participants.  In 
particular,  the  Architect-Engineer  firms  and  the  Decision  Makers  (HRPWC ’s)  think 
the  process  is  a  great  inqrrovement  over  the  previous  MILCON  process.  Most  of  the 
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A-E’s  reported  that  the  process  enhances  their  abihty  to  design  the  project  to  meet  the 
customer’s  requirements  while  still  staying  within  budget.  The  HRPWC’s  seem  to  like 
their  early  involvement  in  the  design  process  and  the  overall  preliminary  design  results. 
The  USAF  Project  Managers  stated  that  Project  Definition  is  especially  effective  in 
identifying  environmental  assessments  and  environmental  design  that  must  be 
conducted  before  construction  begins. 

4.5.2  BCE  Difficulty  with  Process 

For  Base  Civil  Engineering  personnel,  who  must  conduct  on-site  project 
programming  and  host  Project  Definition  and  the  charrette,  the  transition  to  the  new 
process  seems  to  be  somewhat  difficult.  Although  most  agreed  that  the  process  is 
good  (see  Section  4.4),  if  project  partic^ants  do  not  fully  perform  their  tasks,  the 
process  can  break  down  and  leave  the  BCE  with  a  Facility  User  and  HRPWC  who  are 
not  satisfied. 

4.5.3  Preparation  for  Project  Definition 

Several  comments  referred  to  the  lack  of  pre-charrette  preparation  by  the 
project  participants,  in  particular  the  User  and  HRPWC.  In  some  cases,  the  User  and 
HRPWC  were  \ininformed  or  misunderstood  what  was  required  of  them  during  the 
charrette.  It  is  especially  in}portant  for  these  two  groups  to  know  extensively  their 
organization’s  design  requirements  prior  to  the  start  of  the  charrette.  Also,  there 
seems  to  be  some  lack  of  understanding  as  to  what  the  process  involves — ^week-long 
meetings  with  engineers  and  architects  to  develop  a  preliminary  design  solution. 

4.5.4  Team  Commitment/Authority  Delegated 

Several  comments  dealt  with  the  fact  that  individuals  must  be  assigned  to  the 
Project  Management  Team  early  in  the  process,  and  these  individuals  must  remain 
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assigned  to  the  team  at  least  through  the  design  phase,  if  not  the  entire  project.  When 
individual  team  positions,  such  as  the  User  representative  or  HRPWC,  change  over  the 
course  of  the  process,  breakdowns  occur  and  design  changes  increase.  Also,  those 
individuals  asagned  as  team  members  must  be  delegated  the  authority  to  make 
decisions  dining  the  process,  or  the  dynamic  effect  of  having  on-site  analysis  and 
design  meetings  (charrettes)  is  lost. 

4.5.5  Programmed  Amount  Drives  Process 

Several  participants  commented  that  there  needs  to  be  more  flexibility  with  the 
project  budget  (i.e..  Programmed  Amount)  during  Project  Definition.  However,  one 
of  the  primary  reasons  the  Air  Force  initiated  the  new  MILCON  process  was  to  keep 
project  design  and  construction  costs  wdthin  the  estabhshed  budget.  Some  participants 
suggested  that  more  lenient  cost  estimates  be  made  wiien  the  project  budget  is 
established.  Other  participants  suggested  that  a  firm  budget  not  be  established  until 
Project  Definition  is  executed. 
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5.0  Conclusions 


5.1  Effect  of  Project  Definition  on  Design  Success 

Although  a  study  of  only  twenty-five  projects  can  not  be  considered  all- 
conchi^e,  this  study  did  show  that  a  strong,  positive  correlation  exists  between  the 
level  of  Project  Definition  effort  on  Air  Force  construction  projects  and  the  success  of 
the  design  effort  on  those  projects.  The  research  sJiowed  that  on  projects  wdiere  the 
Architect-Engineer  or  in-house  designer  conducted  thorough  on-site  data  gathering, 
on-site  analysis,  and  conducted  on-site  preliminary  design  with  the  assistance  of  the 
Project  Management  Team,  more  successful  designs  were  realized.  For  the  Air  Force, 
design  success  is  defined  as  little  variation  between  cost  estimates  and  actual  costs, 
httle  variation  from  the  planned  schedule,  few  if  any  design  changes,  low  design  costs, 
and  a  customer  that  is  satisfied  with  the  facility  provided  to  his  or  her  organization. 
When  the  new  MILCON  process  is  fully  and  effectively  applied,  money  and  time  are 
saved,  and  the  facility  users  and  base  commanders  are  more  satisfied  with  the  facility 
and  the  construction  process. 

5.2  Effect  of  Charrettes  on  the  Project  Definition  Process 

The  research  showed  that  conducting  charrettes  has  a  positive  effect  on  the 
Project  Definition  and  Design  phases.  In  every  metric  measured  for  Design  Success, 
those  projects  that  had  charrettes  performed  better  than  those  projects  that  did  not 
have  charrettes.  However,  if  the  charrette  process  is  to  be  con^letely  successful  the 
team  members  must  be  thoroughly  prepared  to  discuss  design  requirements  and  all 
team  members  must  play  an  active  role  in  the  process.  Also,  team  members  should  be 
assigned  early  in  the  project,  stay  on  the  team  throughout  the  design  phase,  and  be 
given  the  authority  to  make  design  decisions  for  the  organizations  they  represent. 
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5.3  Customer  Satisfaction  with  the  Project  Definition  Process 

From  the  146  questionnaires  returned  for  inclu^on  in  this  study,  it  is  apparent 
that  the  project  participants  were,  for  the  most  part,  very  satisfied  with  the  Project 
Definition  process.  The  average  rating  on  the  question  specifically  asking  the 
participant’s  opinion  of  the  process  was  7.66  on  a  scale  of  1  to  10,  with  10  being  the 
highest.  Additionally,  the  average  index  on  the  Customer  Satisfaction  Sub-Matrix  was 
812  on  a  scale  from  0  to  1000,  with  1000  being  the  highest.  Having  the  facility  user 
and  project  decision  maker  involved  in  the  development  of  a  preliminary  design  not 
only  improves  the  process  by  reducing  changes,  it  also  inqjroves  the  users’  satisfaction 
with  the  product  designed  and  with  the  entire  process. 
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6.0  Recommendations 


6.1  Actions  Based  on  this  Research 

Based  on  this  research,  the  following  three  actions  are  recommended  for 
AFCEE: 

1.  Continue  to  Use  the  Project  Definition  Process.  This  study  has  shown 
strong  evidence  that  the  new  MILCON  process  produces  successful  design  efforts.  As 
one  questionnaire  respondent  said,  “The  Project  Manager’s  Guide  to  Project 
Definition  is  a  conq)rehensive,  common  sense  approach  to  architectural  programming 
and  concept  design.”  Having  the  facility  user  and  decision  maker  involved  in 
preliminary  planning  is  critical  to  the  successful  design  of  any  &cility.  Additionally, 
having  all  interested  parties  involved  in  fece-to-6ce  charrette  meetings  builds 
teamwork  and  improves  overall  project  performance. 

2.  Improve  Process  Awareness  and  Pre-Charrette  Preparation.  As  shown 
from  the  comments  and  suggestions  received  from  the  participating  project  groups 
(PM,  A-E,  BCE,  Agent,  User,  HRPWC),  the  Air  Force  needs  to  focus  more  effort  in 
preparation  for  the  charrette.  Specifically,  the  Facility  User  and  HRPWC  need  to 
know  explicitly  what  their  organizations’  design  requirements  are.  Also,  more  effort 
needs  to  be  placed  on  educating  all  project  participants  on  what  the  process  involves, 
when  the  charrette  meetings  will  take  place,  and  vshat  is  e^qpected  of  each  individual. 
When  the  User  does  not  know  his  or  her  organization’s  requirements  or  when  various 
individuals  do  not  fully  participate  in  the  process,  the  effect  of  having  face-to-face 
meetings  is  lost. 

3.  Continue  to  Monitor  Project  Definition.  AFCEE  needs  to  monitor 
Project  Definition  by  using  the  Design  Success  Matrix  used  in  this  study  or  some  other 
form  of  rating  as  a  benchmark  of  how  a  project  is  performing  and  what  needs  to  be 
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improved  in  order  to  have  successful  design  and  construction  efiforts  on  that  project. 
As  shown  from  this  study,  the  level  of  Project  Definition  effort  can  predict  the  level  of 
design  success.  The  Construction  Industry  Institute  (Cn)  is  conducting  research  into 
the  development  of  a  Project  Definition  Rating  Index  for  General  Building,  and  CE 
has  Predictive  Tools  computer  software  that  could  aid  AFCEE  and  other  Major 
Commands  in  monitoring  the  early  phases  of  their  construction  projects. 

6.2  Actions  Based  on  General  Observations 

Enroll  the  Air  Force  in  the  Construction  Industry  Institute.  For  the  past 
twelve  years,  CD  has  conducted  extensive  research  and  produced  cost  saving 
initiatives  for  the  entire  construction  process.  A  few  of  the  areas  studied  include  Pre- 
Project  Planning,  Design,  QuaUty  Management,  Safety,  Project  Team  Building, 
Constructabihty,  Electronic  Data  Management,  and  Partnering.  These  research 
initiatives  benefit  not  only  construction  contractors  but  owner  organizations  as  well. 
The  U.S.  Army  Corps  of  Engineers  and  the  U.S.  Naval  Facilities  Engineering 
Command  have  both  joined  CE  with  very  positive  results.  CE  contains  a  wealth  of 
construction  information  and  the  Air  Force  and  AFCEE  need  to  take  advantage  of  that 
information. 

6.3  Future  Research 

From  this  study,  the  following  areas  of  further  research  are  recommended: 

1.  Project  Definition  versus  Construction  Success.  This  study  showed  that 
Project  Definition  has  a  positive  correlation  with  the  level  of  design  success.  An 
additional  investigation  should  be  conducted  to  study  the  effect  Project  Definition  has 
on  the  outcome  of  the  entire  project.  It  may  be  possible  to  study  the  same  projects 
used  in  this  research. 
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2.  Development  of  a  Predictive  Tool  Specifically  for  the  Air  Force 
Construction  Process.  As  mentioned  in  the  previous  section,  the  Air  Force  should 
continue  to  rate  its  Project  Definition  eflforts  as  a  means  of  monitoring  the  early  stages 
of  its  projects  in  order  to  predict  the  level  of  overall  project  success.  The  tool  should 
be  relatively  single  to  use  and  produce  some  sort  of  numerical  rating. 

3.  Feasibilityofthe  Air  Force  or  AFCEE  Joining  Cn.  As  discussed 
previously,  the  benefits  of  joining  Cn  are  extensive.  A  study  could  be  conducted  to 
show  the  benefits  versus  the  costs  of  membership. 
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Appendix  1 

Air  Force  Project  Manager  (DM/CM) 

This  questionnaire  will  be  used  to  analyze  and  in^rove  the  Project  Definition  process 
on  U.S.  Air  Force  military  construction  projects.  It  is  inqiortant  that  you  provide 
accurate  information.  If  you  are  uncertain  about  any  particular  question  or  response, 
please  review  the  Project  Manager’s  Guide  to  Project  Definition  or  the  Project 
Definition  Tutor.  On  projects  where  AFCEE  is  also  the  DA/CA  please  con^lete  the 
Design  Agent  survey  afl;er  couqileting  the  DM/CM  survey 

Project  Name: _ _ _ _ _ 

Project  Location  (Base): _ _ _ _ _ 

Interviewee  Name: _ _  Date: _ 

Organization:  _ _ Phone: _ Fax: _ 

Please  check  all  appropriate  responses  for  the  project  listed  above. 

General  Project  Information 

1.  Project  Delivery  Strategy: 

□  Design-Bid-Build  □  Design-Build  □  Turnkey 

□  Other _ _ 


2.  Design  Criteria: 

□  Traditional  Design  □  Site  Adapt  □  Defimtive  Guide  (e.g.  Dorm  Guide) 

□  Other _ _ 


3.  DM/CM: 

□  AFCEE  DM  □  AFCEE  CM 

4.  Design  Agent: 

□  Corps  of  Engineers  □  NAVFAC  □  Air  Force 

5.  Which  of  the  following  (if  any)  were  used  on  this  project? 

□  charrettes  □  on-site  data  gathering  □  on-site  analysis  □  on-site  design 
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Please  fill  in  the  blank. 

Project  Cost  Information 

1.  Please  provide  the  CWE  of  the  Parametric  Cost  Estimate  that  was  developed 

during  Project  Definition.  $ _  • 

2.  Please  provide  the  CWE  of  the  A-E/Designer’s  cost  estinaate  provided  to  you  when 

the  design  was  100%  complete.  $ _ . 

3.  Please  provide  the  awarded  contractor’s  cost  for  this  project  (i.e.  Low  Bidder’s 

CWE).$ _ . 

4.  Please  provide  the  amount  for  >^4iich  this  project  was  programmed  by  the 

MAJCOM  (i.e.  Programmed  Amount).  $ _ ^ 

5.  How  many  design  changes  resulting  fi'om  approved  comments  were  issued  to  the 

A-E/Designer  after  Project  Definition  during  the  design  of  this  project? _ 

6.  What  was  the  cost  to  the  Air  Force  for  the  design  changes  ordered  during  the 

design  phase  (fi’om  PD  to  Award)  for  this  project?  $ _ (A-E  cost) 

$ _ (in-house  cost) 

7.  What  was  the  total  design  time  required  for  this  project? 

a.  fiom  A-E  Notice  to  Proceed  to  Construction  Award _ _ 

b.  fiom  Project  Definition  to  Construction  Award _ 

8.  How  much  time  was  added  to  the  design  schedule  due  to  changes  after  Project 

Definition?  _ 


9.  What  was  the  cost  to  the  Air  Force  for  the  entire  Project  Definition  phase? 

$ _ ^(A-Ecost)  $ _ ^(in-house  cost) 

10.  What  was  the  cost  to  the  Air  Force  to  conq)lete  the  design  phase  fiom  PD  to 

Award?  $ _ ^(A-Ecost)  $ _ ^(in-house  cost) 
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For  each  of  the  following  statements,  please  circle  the  response  (1-10)  which  best 
represents  your  agreement  or  disagreement  with  the  statement.  Please  fill  in  the 
blanks  for  the  follow-on  questions. 


Project  Management  Team 


1.  The  A-E  firm/in-house  design  team  chosen  to  conduct  the  charrettes  during  the 
Project  Definition  phase  of  this  project  was  well  prepared  and  effectively  led  the  on¬ 
site  analysis  and  design  required  by  the  contract. 


Strongly 

agree 


agree 


sli^tly 

agree 


neutral 


sli^tly 

disagree 


disagree 


10  9 . 8 . 7 . 6 . 5  . 4 . 3 . 2  - 


Strongly 

disagree 

1 . 0 


2a.  The  Cost  Estimator  used  by  the  A-E/Designer  to  prepare  the  parametric  cost 
estimate  performed  well. 


strongly 

agree 


10  -  -*  9 


slightly  ,  slightly 

agree  disagree 

•8 . 7 . 6 . 5  . 4 . 3 


strongly 

disagree  disagree 

•2 . 1 . 0 


2b.  The  Cost  Estimator  used  by  the  A-E/Designer  to  prepare  the  parametric  cost 
estimate  provided  on-site  cost  information  during  the  charrette  for  alternative  design 
decisions. 


Strongly 

agree 


10 


slightly  ,  subtly 

agree  disagree 

9 . 8 . 7 . 6 . 5  . 4 . 3 


strongly 

disagree  disagree 

■2 . 1  0 


3.  The  Design  Agent  (U.S.  Army  Corps  of  Engineers,  U,S.  NAVFAC,  or  U.S.  Air 
Force)  contributed  greatly  to  the  successful  corr^letion  of  the  Project  Defimtion 


process. 


strongly 

agree 


10----  9 


slightly  ,  slightly 
agre®  agree  neutral  disagree 

-8 . 7 . 6 . 5  . 4 . 3 


Strongly 

disagree  disagree 

-2 . 1 . 0 


4.  The  Facility  End  User/Future  Occupant  was  actively  involved  and  effectively 
communicated  his/her  unit’s  design  requirements  to  the  A-E  during  the  Project 
Definition  phase  of  this  project. 


strongly 

agree 


10 


9 


sli^tly  .  subtly 

sg*"®®  agree  “®'dral  disagree 

-8 . 7 . 6 . 5 . 4 . 3 


strongly 

disagree 

■■■  2 . 1 . 0 


5.  The  Highest  Ranldng  Person  With  Concern  (HRPWC)/Decision  Maker  for  this 
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project  was  effectively  involved  in  the  decision  making  process  during  the  Project 
Definition  phase. 


strongly 

agree 


10  9 


sli^tly  ,  slightly 

agree  disagree 

'8 . 7  ■•6 . 5  . 4 


strongly 

disagree  disagree 

3 . 2 . 1 . 0 


6.  The  Base  Civil  Engineering  staff  responsible  for  this  project  helped  solve  problems, 
effectively  supported  the  designer,  and  contributed  adequate  mformation  during  the 
Project  Definition  phase. 


strongly 

agree 


agree 


subtly 

agree 


neutral 


sli^tly 

disagree 


disagree 


strongly 

disagree 


10  •••■  9 . 8 . 7 . 6 . 5 . 4 . 3 . 2 . 1 . 0 


7.  The  Major  Command  (MAJCOM)  responsible  for  this  project  helped  solve 
problems  and  provided  adequate  guidance  during  the  Project  Defimtion  phase. 


strongly 

agree 


agree 


sli^tly 

agree 


neutral 


sli^tly 

disagree 


disagree 


strongly 

disagree 


10  ■  9 . 8 . 7 . 6 . 5  . 4 . 3 . 2 . 1 . 0 


8.  The  Base  Support  Team  (Fire  Dept.,  Comm.,  Safety,  etc.)  provided  early,  timely, 
and  adequate  design  requirements  during  the  Project  Definition  phase. 


Strongly 

agree 


agree 


slightly 

agree 


neutral 


sli^tly 

disagree 


disagree 


strongly 

disagree 


10-  -  9 . 8- •••  7 . 6 . 5  . 4 . 3 . 2 . 1 . 0 


9.  The  Outside  Agencies  (EPA,  Utilities,  Historical,  etc.)  involved  with  this  project 
provided  early,  timely,  and  adequate  design  requirements  during  the  Project  Definition 
phase. 


Strongly 

agree 


10  •  9 


slightly  ,  subtly 

agree  “®“*ral  disagree 

-  8 . 7 . 6 . 5  . 4  ••• 


strongly 

disagree  disagree 

3 . 2 . 1 . 0 


10.  As  the  Air  Force  Project  Manager,  I  feel  I  was  effective  in  bringing  the  Project 
Management  Team  to  consensus. 


strongly 

agree 


10  9 


sli^tly  ,  sli^tly 

agree  disagree 

■8 . 7 . 6 . 5  . 4  ■ 


strongly 

disagree  disagree 

3 . 2 . 1 . 0 


NOTE:  If  you  did  not  manage  this  project  during  Project  Definition,  still  evaluate 
your  predecessor. 

not  effective,  wliat  prevented  you  or  your  predecessor  fi-om  being  effective? 
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11.  I  feel  the  right  people  were  involved  in  die  Project  Definition  process. 


agree  neutral 

agree  ^  agree 

10-  •  9 . 8 . 7- . 6 . 5  • 

If  not,  vriio  should  have  been  more  involved? 


sliiJitly 
neutral  disagree 


disagree 


strongly 

disagree 

--  0 


12.  I  was  satisfied  with  the  Project  Definition  process,  because  I  believe  the  process 
enhanced  the  final  product. 


strongly 

agree 

10  ■■■  9 


sli^tly 


sli^tly 
neutral  disagree 

5  . 4 . 3 


disagree 


strongly 

disagree 


2 . 1 . 0 


Project  Definition  Products 

1.  The  Project  Management  Plan  developed  for  this  project  effectively  communicated 
strategic  decisions  and  the  project  management  team’s  membership,  roles,  and 
responsibilities. 


strongly 

agree 

10-  -  9 


sli^tly  sli^tly 

agree  disagree 

--  6 . 5  . 4  --' 


disagree 


strongly 

disagree 


2.  At  the  Pre-Definition  Conference,  the  project  scope,  customer  requirements.  Base 
CoEDprehensive  Plan  (BCP),  base  standards,  and  environmental  issues  were  reviewed 
with  the  A-E/Designer,  and  all  project  team  member  responsibilities  were  defined. 


Strongly 

agree 

10 


subtly  ,  subtly 

agree  disagree 

•6 . 5  . 4  • 


disagree 


strongly 

disagree 

1 . 0 


3.  The  Data  Gathering  and  Requirements  Analysis  portions  of  the  process  produced 
the  needed  information  to  make  critical  design  decisions  diuing  Project  Defimtion. 

strondv  sli^tly  ,  slightly  stron 

agri  “Sree  neutral  disagree 

10  -  -  9 . 8 . 7 . 6 . 5  . 4 . 3 . 2 . 1 . 0 


strongly 

disagree  disagree 

2 . 1 . 0 


■7 


4.  The  Site  Development  Plan  produced  during  Project  Definition  adequately  depicted 
all  site  requirements;  utility  systems’  sizes,  loads,  and  connection  points;  rough  site 
contours  and  parking;  and  it  contributed  to  vahdating  the  building  support  cost 
(outside  the  5’  line). 


Strongly 

agree 


lo  ----  9 


subtly  _  ,  sUghtly 
*8^®®  agree  neutral  disagree 

•8 . 7 . 6 . 5 . 4 . 3 


Strongly 

disagree  disagree 

-2 . 1 . 0 


5.  The  Schematic  Floor  Plan  developed  during  Project  Definition  satisfied  the  user  by 
adequately  depicting  all  ^ace  requirements  of  the  facility. 


strongly 

agree 


agree 


sli^tly 

agree 


neutral 


sli^tly 

disagree 


disagree 


strongly 

disagree 


10  --  9 . 8 . 7 . 6 . 5  . 4 . 3 . 2 . 1 . 0 


If  the  user  was  not  satisfied,  why  not? 


How  many  alternative  floor  plans  were  developed? 


6.  Design  criteria  and  building  subsystem  (such  as  exterior  wall,  roof^  mechanical, 
electrical,  structural,  communications,  etc. )  narratives  (with  alternatives)  were  fully 
and  accurately  developed  during  Project  Definition. 


Strongly 

agree 


10  ----  9 


sliAtly  ,  sli^tly 

agr®®  agree  n®“tral  disagree 

8 . 7 . 6 . 5  . 4 . 3 


Strongly 

disagree  disagree 


7.  The  Building  Elevations  produced  dining  Project  Definition  complied  with  base 
architectural  standards  and  the  BCP. 


Strongly 

agree 


10-'  -  9 


sli#itly  ,  subtly 

sgr®®  agree  disagree 

■8 . 7 . 6 . 5  . 4  - 


strongly 

disagree  disagree 

3 . 2 . 1 . 0 


8.  The  Parametric  Cost  Estimate  developed  during  the  charrette  was  accurate. 


strongly 

agree 


10  ■  9 


sliAtly  ,  sli^tly 

agree  “®“t*'al  disagree 

■8 . 7---  6 . 5  . 4 . 3 


Strongly 

disagree  disagree 

•2 . 1 . 0 


9.  The  Environmental  Checklist  developed  during  Project  Definition  alerted  the 
Project  Management  Team  to  the  existence  of  critical  environmental  issues. 


Strongly 

agree 


lO  ---  9 


slightly  ,  slightly 

agr®®  agree  “®'*tral  disagree 

■8 . 7 . 6 . 5  . 4 . 3 


strongly 

disagree  disagree 

2 . 1 . 0 
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10.  Please  provide  any  comments  or  suggestions  you  have  on  hrqproving  the  Project 
Definition  process.  Your  frank  and  honest  input  to  this  study  is  critical. 


Thank  you  for  your  cooperation  in  filling  out  this  questionnaire.  If  you  are  also  the 
DA/CA,  please  complete  the  Design  Agent  survey  fisted  hereafter. 
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Appendix  2 

Architect-Engineer  (or  In-House  Designer) 


Project  Name: _ _ 

Project  Location  (Base): _ _ _ _ 

Interviewee  Name: _ _  Date: 


Company: _ _  Phone: _  Fax: _ 

For  each  of  the  following  statements,  please  circle  the  response  (1-10)  which  best 
represents  your  agreement  or  disagreement  with  the  statement.  Please  fill  in  the 
blanks  for  the  follow-on  questions. 


1.  The  facility  user’s  requirements  were  clearly  defined  during  Project  Definition. 

strondv  sliAtly  .  sli^tly  strongly 

agree  agree  disagree  disagree  disagree 

10  ■  9 . 8 . 7 . 6 . 5  4 . 3 . 2 . 1 . 0 

2.  The  project  goals  and  requirements  were  identified  and  vahdated  before  design 
started. 

stronelv  sliAtly  ,  sli^tly  strongly 

a“£  a?ee  disagree  disagree 

10-  -  9 . 8 . 7 . 6 . 5  4 . 3 . 2 . 1 . 0 

3.  The  Project  Definition  process  was  adequately  communicated  to  my  firm/designers. 

stronelv  sliAtly  ,  sli^tly  strongly 

agree  agree  “®*d*'®*  disagree  disagree  disagree 

10  -  9 . 8 . 7 . 6 . 5  4 . 3 . 2 . 1 . 0 

4.  During  Project  Definition  I  was  able  to  discuss  and  present  all  feasible  design 
alternatives  (floor  plans,  site  plans,  etc.). 

stronelv  sliAtly  ,  sli^tly 

apS  “Sree  a^ee  disagree  disagree 

10-  -  9 . 8 . 7 . 6 . 5  4 . 3 . 2 .  1 

5.  A  consensus  was  reached  among  project  team  members  on  selected  design 

alternatives  and  building  sub-systems. 

strongly  slightly  ^  ,  slightly 

agr^  agree  disagree 

10-  •  9 . 8 . 7 . 6 . 5  4 . 3 . 2 . 1 


Strongly 

disagree 

0 


strongly 

disagree 

-  0 
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6.  The  right  people  (facility  user,  decision  maker,  base  support,  outside  agencies,  etc.) 
were  involved  in  the  Project  Definition  process. 


strongly 

agree 


10  9 


sli^tly  ,  sli^tly 

agree  agree  n®'*tral  disagree 

■8 . 7 . 6 . 5  . 4-- 


Strongly 

disagree  disagree 

3 . 2 . 1 . 0 


If  not,  who  should  have  been  involved  or  more  involved? 


7.  The  charrette(s)  provided  a  design  solution: 
a.  To  a  satisfied  customer, 


Strongly 

agree 

agree 

sligjitly 

agree 

neutral 

sli^tly 

disagree 

disagree 

10  - 9  ••• 

■  8  - 

...7.. 

. 6  ■■ 

...5 ... 

■4 . 

3  • 

. 2 

1 

b.  In  less  time. 

Strongly 

agree 

agree 

sli^tly 

agree 

neutral 

sligjitly 

disagree 

disagree 

10  9  ••• 

••  8  ••• 

...7.. 

. 6 

...5 

--4 . 

3  • 

. 2  ••• 

1 

c.  At  a  lower  cost. 

strongly 

agree 

agree 

sli^tly 

agree 

neutral 

sligjitly 

disagree 

disagree 

10  9 

..  8  .... 

..,7.. 

6 

...  5  ... 

■--'4 . 

3  ■ 

. 2  - 

1 

Strongly 

disagree 

■  0 


strongly 

disagree 

--  0 


strongly 

disagree 

0 


8.  The  ‘Library  Wall’,  if  used,  was  an  eflFective  communication  technique  in  bringing 
the  team  to  a  consensus. 


Strongly 

agree 


10 -••• 


slightly  ,  slightly  ..  strongly 

agree  ipee  n®“^al  disagree  disagree  disagree 

9 . 8 . 7 . 6 . 5 . 4 . 3 . 2 . 1 . 0--NA 


NOTE:  The  Library  Wall  is  a  tenq)orary  wall  display  of  information  that  relates  to 
the  project. 


9.  How  much  time  was  added  to  the  design  schedule  due  to  Air  Force  changes  after 
Project  Definition?_ _ _ _ _ 


10.  I  was  satisfied  with  the  Project  Definition  process,  because  I  believe  the  process 
enhanced  the  final  product. 


strongly 

agree 


10  •••• 


subtly 

*8^®®  agree  "®'*®' 

9 . 8 . 7 . 6 . 5  •• 


slightly  strongly 

disagree  disagree  disagree 

•4 . 3  •••••2 . 1 . 0 
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1 1.  Please  provide  any  comments  or  suggestions  you  have  on  mq)roving  the  Air 
Force’s  Project  Definition  process.  Your  frank  and  honest  input  to  this  study  is 
critical. 


Thank  you  for  your  cooperation  in  filling  out  this  questionnaire.  Please  enclose  the 
conq)leted  questionnaire  in  the  envelope  provided  and  mail  it  by  19  Jul  95. 


53 


Appendix  3 

Base  Civil  Engineering  Staff 


Project  Name: _ 

Project  Location  (Base): 


Interviewee  Name: 


Date: 


Organization: _ Phone: _  Fax: _ 

For  each  of  the  following  statements,  please  circle  the  response  (1-10)  which  best 
represents  your  agreement  or  disagreement  with  the  statement.  Please  fill  in  the 
blanks  for  the  follow-on  questions. 


1. 


The  facility  user’s  requirements  were  clearly  defined  during  Project  Defimtion. 


strongly 

agree 


agree 


sligjitly 

agree 


neutral 


sli^tly 

disagree 


disagree 


strongly 

disagree 


10-  -  9 . 8 . 7 . 6 . 5  . 4 . 3 . 2 . 1 . 0 


2.  The  project  goals  and  requirements  were  identified  and  validated  before  design 
started. 


Strongly 

agree 


10-  ’  9 


sliAtly  .  subtly  strongly 

agree  neutral  disagree  disagree  disagree 

-8 . 7 . 6 . 5  . 4 . 3 . 2 . 1 . 0 


3. 


The  Project  Definition  process  was  adequately  communicated  to  my  organization. 


Strongly 

agree 


10  -  9 


agree 

-8 . 7 


sli^tly 

agree 


■6 


sligiitly  strongly 

neutral  disagree  disagree  disagree 

5  . 4 . 3 . 2 . 1 . 0 


4.  All  feasible  design  alternatives  (floor  plans,  site  plans,  etc.)  were  considered  during 
Project  Definition. 


strongly 

agree 


agree 


slightly 

agree 


neutral 


sligjhtly 

disagree 


disagree 


strongly 

disagree 


10  9 . 8 . 7 . 6 . 5  . 4 . 3 . 2 . 1 . 0 


54 


5a.  A  consensus  was  reached  among  project  team  members  on  selected  design 
alternatives  and  building  sub- systems. 


Strongly 

agree 


10  •  9 


slightly  ,  subtly 

agree  disagree 

-  8 . 7 . 6 . 5  . 4  ••• 


Strongly 

disagree  disagree 

3 . 2 . 1 . 0 


5b.  The  Base  Support  Team  (Fire  Dept.,  Comm,  Safety,  etc.)  provided  early,  timely, 
and  adequate  design  requirements  during  the  Project  Definition  phase  of  this  project. 


Strongly 

agree 


agree 


sli^tly  ,  subtly 

agree  disagree 


disagree 


strongly 

disagree 


10-  -  9 . 8 . 7 . 6 . 5  . 4 . 3 . 2 . 1 . 0 


6.  The  right  people  (facility  user,  decision  maker,  base  support,  outside  agencies,  etc.) 
were  involved  in  the  Project  Definition  process. 


Strongly 

agree 

10-  '  9 


subtly  ,  slightly 
agree  “®t*tral  disagree 

■8 . 7 . 6 . 5  ■■■4 . 3 


Strongly 

disagree 

■2 . 1 . 0 


If  not,  who  should  have  been  involved  or  more  involved? 


7.  The  charrette(s)  provided  a  design  solution; 
a.  To  a  satisfied  customer. 


strongly 

agree 

agree 

sli^tly 

agree 

neutral 

subtly 

disagree 

disagree 

strongly 

disagree 

10 

.  9  ... 

-8 

,..7.. 

6  - 

..  5 .. 

■  4 . 

3 . 

2 . 1 . 0 

b.  In  less  time. 

Strongly 

agree 

agree 

sli^tly 

agree 

neutral 

sli^tly 

disagree 

disagree 

strongly 

disagree 

10 

.  9  ... 

8- 

•  7  -■ 

. 6  • 

...5  .. 

••  -4 . 

3  - 

. 2 . 1 . 0 

c.  At  a  lower  cost. 

Strongly 

agree 

agree 

sli^tly 

agree 

neutral 

sligjitly 

disagree 

disagree 

strongly 

disagree 

10- 

.  9  ... 

...g.... 

.7., 

...5... 

..  5  .. 

4 . 

..3 . 

. 2 . 1 . 0 

8.  The  ‘Library  Wall’,  if  used,  was  an  effective  communication  technique  in  bringing 
the  team  to  a  consensus. 


Strongly 

agree 

10-..- 


sliAtly  ,  shortly  strongly 

^8^®®  agree  disagree  disagree  disagree 

9 . 8 . 7 . 6 . 5 . 4 . 3 . 2 . 1 . 0--  NA 


NOTE:  The  Library  Wall  is  a  temporary  wall  display  of  information  that  relates  to 
the  project. 
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9.  Compared  with  other  facilities  of  this  type,  this  facility  will  be  easy  (physically  and 
economically)  to  maintain. 


strongly 

agree 


10-..  9 


sUgihtly  ,  slightly 

*8^®®  agree  “®'*'®‘  disagree 

•8 . 7 . 6 . 5  . 4  - 


disagree 

3 . 2 


strongly 

disagree 

1 . 0 


10.  This  fecility  will  fit  in  well  with  the  architectural  scheme  of  the  base. 


strongly 

agree 


10  --  9 


sli^tly  ,  slightly 

agree  agree  disagree 

-8 . 7 . 6 . 5  . 4’  - 


disagree 

3 . 2 . 


strongly 

disagree 

--  0 


1 1.  I  was  satisfied  with  the  Project  Definition  process,  because  I  believe  the  process 


enhanced  the  final  product. 


Strongly 

agree 


10--.-  9 


agree 

8 . 7 


sligjitly 
agree 
.  6-- 


neutral 

...  5  .. 


slightly 

disagree 

...4... 


strongly 

disagree  disagree 

3 . 2 . 1 . 0 


12.  Please  provide  any  comments  or  suggestions  you  have  on  in5)rovmg  the  Air 
Force’s  Project  Definition  process.  Your  frank  and  honest  input  to  this  study  is 
critical. 


Thank  vou  for  your  cooperation  in  filling  out  this  questionnaire.  Please  enclose  the 
completed  questionnaire  in  the  envelope  provided  and  mail  it  by  19  Jul  95. 
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Appendix  4 


Design  Agent 

(Corps  of  Engineers,  NAVFAC,  or  Air  Force) 


Project  Name: _ _ 

Project  Location  (Base): _ 

Interviewee  Name: _ _  Date: 


Organization: _ _  Phone: _  Fax: _ 

For  each  of  the  following  statements,  please  circle  the  response  (1-10)  which  best 
represents  your  agreement  or  disagreement  with  the  statement.  Please  fill  in  the 
blanks  for  the  follow-on  questions. 


1. 


The  facihty  user’s  requirements  were  clearly  defined  during  Project  Definition. 


strongly 

agree 

10  9 


slisjitly  ,  sli^tly 

*8"®®  agree  disagree 

-8 . 7 . 6 . 5  . 4 . 3 


strongly 

disagree  disagree 

'  2 . 1 . 0 


2.  The  project  goals  and  requirements  were  identified  and  validated  before  design 
started. 


3. 


strongly 

agree 


10----  9 


sUAtly  subtly  strongly 

®8^®®  agree  “^^dral  disagree  disagree  disagree 

■8 . 7 . 6 . 5  . 4 . 3 . 2 . 1 . 0 


The  Project  Definition  process  was  adequately  commxmicated  to  my  organization. 


Strongly 

agree 


10  ••  9 


slightly  .  sli^tly 

agree  o®'t^tal  disagree 

■8 . 7 . 6 . 5 . 4 . 3 


Strongly 

disagree  disagree 

-2 . 1 . 0 


4.  All  feasible  design  alternatives  (floor  plans,  ^e  plans,  etc.)  were  considered  dming 
Project  Definition. 


strongly 

agree 


agree 


subtly  ,  subtly 

agree  disagree 


disagree 


strongly 

disagree 


10-  -  9 . 8 . 7 . 6 . 5  . 4 . 3 . 2 . 1 . 0 

5.  A  consensus  was  reached  among  project  team  members  on  selected  design 
alternatives  and  building  sub- systems. 


Strongly 

agree 


10----  9 


sliAtly  ,  sli^tly 

agree  t*®****"®*  disagree 

■8 . 7 . 6 . 5  . 4-- 


Strongly 

disagree  di,3g,gg 

3 . 2 . 1 . 0 
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6.  The  right  people  (facility  user,  decision  maker,  base  support,  outside  agencies,  etc.) 
were  involved  in  the  Project  Definition  process. 


Strongly 

agree 


10  --  9 


subtly  ,  subtly 

agree  disagree 

■8 . 7 . 6 . 5  . 4 . 3 


strongly 

disagree 

-  2 . 1 . 0 


If  not,  who  should  have  been  involved  or  more  involved? 


7.  The  charrette(s)  provided  a  design  solution; 
a.  To  a  satisfied  customer. 


strongly 

agree 

agree 

slightly 

agree 

neutral 

sli^tly 

disagree 

disagree 

strongly 

disagree 

10  ••••  9  ••• 

••  8  - 

••■7- 

. 6  - 

••5  ••• 

••■4 . 

3- 

. 2 

1 

. 0 

b.  In  less  time. 

Strongly 

agree 

agree 

sligjhtly 

agree 

neutral 

slightly 

disagree 

disagree 

strongly 

disagree 

10  -  9  -- 

•  8- 

-■  7- 

. 6 

•  5  • 

-  4 . 

3  ■ 

. 2 

■  1 

. 0 

c.  At  a  lower  cost. 

Strongly 

agree 

agree 

slightly 

agree 

neutral 

slightly 

disagree 

disagree 

strongly 

disagree 

10  ■■■  9 

8 

--■7- 

. 6  ■ 

•5  - 

-4 . 

3  ■ 

. 2  -■■■ 

1 

. 0 

8.  The  ‘Library  Wall’  if  used,  was  an  effective  commimication  technique  in  bringing 
the  team  to  a  consensus. 


Strongly 

agree 


agree 


slightly  ,  slightly 

agree  disagree 


disagree 


strongly 

disagree 


10  ■■■■9 . 8 . 7 . 6 . 5 . 4 . 3 . 2 . 1 . 0--NA 

NOTE:  The  Library  Wall  is  a  temporary  wall  display  of  information  that  relates  to 
the  project. 


9.  I  was  satisfied  with  the  Project  Definition  process,  because  I  beUeve  the  process 
enhanced  the  final  product. 


Strongly 

agree 


agree 


sli^tly 

agree 


neutral 


sli^tly 

disagree 


disagree 


strongly 

disagree 


10-  •  9 . 8 . 7 . 6 . 5  . 4 . 3 . 2 . 1 . 0 
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10.  Please  provide  any  comments  or  suggestions  you  have  on  mq)roving  the  Air 
Force’s  Project  Definition  process.  Your  fi-ank  and  honest  input  to  this  study  is 
critical. 


Thank  you  for  yom  cooperation  in  filling  out  this  questionnaire.  Please  enclose  the 
conqjleted  questionnaire  in  the  envelope  provided  and  mail  it  by  19  Jul  95. 
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Appendix  5 
Facility  User 


Project  Name: _ _ _ 

Project  Location  (Base): _ 

Interviewee  Name: _ _ Date: 


Organization: 


Phone: 


Fax: 


For  each  of  the  following  statements,  please  circle  the  response  (1-10)  which  best 
represents  your  agreement  or  disagreement  with  the  statement.  Please  fill  in  the 
blanks  for  the  follow-on  questions. 

1.  My  organization’s  requirements  were  clearly  defined  during  Project  Definition. 

strongly  sli^tly  ,  sUgJitly  strongly 

agree  agree  disagree  disagree  disagree 

10-  -  9 . 8 . 7 . 6 . 5  . 4 . 3 . 2 . 1 . 0 

2.  The  project  goals  and  requirements  were  identified  and  validated  before  design 
started. 

strongly  sU^tly  ,  sUgJitly  strongly 

agrS^  agree  disagree  disagree 

10  -  9 . 8 . 7 . 6 . 5  . 4 . 3 . 2 . 1 . 0 

3.  The  Project  Definition  process  was  adequately  communicated  to  my  organization. 

strongly  sli^tly  ,  sli^tly  strongly 

agri^  agree  disagree  disagree  disagree 

10  •••■  9 . 8 . 7 . 6 . 5  . 4 . 3 . 2 . 1 . 0 

4.  All  feasible  design  alternatives  (floor  plans,  site  plans,  etc.)  were  considered  during 
Project  Definition. 

strongly  sli^tly  sli^tly 

agree  agree  disagree 

10-  -  9 . 8 . 7 . 6 . 5  . 4 . 3 . 2 .  1 

5.  A  consensus  was  reached  among  project  team  members  on  selected  design 
alternatives  and  building  sub-systems. 

strongly  sli^tly  sli^tly 

agr^  agree  disagree 

10-  -  9 . 8 . 7 . 6 . 5  . 4 . 3 . 2 . 1 


disagree 


strongly 

disagree 

...  0 


strongly 

disagree 

0 
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6.  The  right  people  (facility  user,  decision  maker,  base  support,  outside  agencies,  etc.) 
were  involved  in  the  Project  Definition  process. 


strongly 

agree 


10  ----  9 


sUghtly  _  ,  sli^tly 
“S’"®®  agree  “eutral  disagree 

■8 . 7 . 6 . 5  . 4 . 3 


strongly 

disagree  disagree 

-■  2 . 1 . 0 


If  not,  who  should  have  been  involved  or  more  involved? 


7.  The  charrette(s)  provided  a  design  solution; 
a.  To  a  satisfied  customer. 


Strongly 

agree 

agree 

slightly 

agree 

neutral 

sligjitly 

disagree 

disagree 

Strongly 

disagree 

10  9  ■■■ 

8 

...  7  .. 

■  6 

...  5  .. 

■4 . 

3  . 

. 2  ---- 

1 

. 0 

b.  In  less  time. 

Strongly 

agree 

agree 

slightly 

agree 

neutral 

sli^tly 

disagree 

disagree 

strongly 

disagree 

10  •  9 

..  g  .... 

-  7  ■■ 

. 6  ■ 

...  5  .. 

■4 . 

3  . 

. 2  --- 

1 

. 0 

c.  At  a  lower  cost. 

Strongly 

agree 

agree 

sli^tly 

agree 

neutral 

subtly 

disagree 

disagree 

strongly 

disagree 

10-...  9  --- 

..  g  .... 

...  7  .. 

•  6  ■■■ 

...  5  .. 

-  4 

3  - 

. 2 

1 

. 0 

8.  The  ‘Library  Wall’,  if  used,  was  an  effective  communication  technique  in  bringing 
the  team  to  a  consensus. 


Strongly 

agree 


agree 


sli^tly 

agree 


neutral 


subtly 

disagree 


disagree 


strongly 

disagree 


10 --9 . 8 . 7 . 6 . 5 . 4 . 3 . 2 . 1 . 0--NA 

NOTE:  The  Library  Wall  is  a  temporary  wall  display  of  information  that  relates  to 
the  project. 

9.  This  facility  meet  the  mission  needs  of  my  organization. 


Strongly 

agree 


agree 


sli^tly 

agree 


neutral 


sli^tly 

disagree 


disagree 


strongly 

disagree 


10  - •••  9 . 8 . 7 . 6 . 5  . 4 . 3 . 2 . 1 . 0 


10.  This  facility  will  enhance  the  job  effectiveness  of  the  personnel  in  my  organization. 


strongly 

agree 


10  ••  9 


slightly  ,  sli^tly 

agree  disagree 

■8 . 7 . 6 . 5  . 4  - 


strongly 

disagree  disagree 

3 . 2 . 1 . 0 


11.  I  was  satisfied  with  the  Project  Definition  process,  because  I  believe  the  process 
enhanced  the  final  product. 


strongly 

agree 


10-  -  9 


sUAtly  ,  subtly  strongly 

®g'‘®®  agree  disagree  disagree  disagree 

•8 . 7 . 6 . 5  . 4 . 3 . 2 . 1 . 0 
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12.  Please  provide  any  comments  or  suggestions  you  have  on  inq)roving  the  Air 
Force’s  Project  Definition  process.  Your  frank  and  honest  input  to  this  study  is 
critical. 


Thank  you  for  your  cooperation  in  filling  out  this  questionnaire.  Please  enclose  the 
completed  questioimaire  in  the  envelope  provided  and  mail  it  by  19  Jul  95. 
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Appendix  6 

Highest  Ranking  Person  With  Concern/Decision  Maker 


Project  Name; _ 

Project  Location  (Base): 


Interviewee  Name: 


Organization: 


Phone: 


For  each  of  the  following  statements,  please  circle  the  response  (l-IO)  which  best 
represents  your  agreement  or  disagreement  with  the  statement. 


1 .  The  facility  user’s  requirements  were  clearly  defined  during  Project  Defimtion. 


disagree 


strongly 

disagree 


strongly  sligjitly  ,  sli^tly  stron 

a?ee  disagree  disag 

10  •••■  9 . 8 . 7 . 6 . 5  4  - ■•••  3 . 2 . 1 . 0 

2.  This  fecility  will  enhance  the  mission  effectiveness  of  the  base. 

strongly  slightly  slightly  stron 

aT^  ®Sree  neutral  disagree  disag 

10  ■■  9  . 8 . 7 . 6 . 5  4 . 3 . 2 . 1 . 0 

3.  I  was  satisfied  with  my  iovolvement  in  the  final  design  decisions  made  during 
Project  Definition. 

strongly  sU^tly  ,  sU^tly  stron 

agr^  neutral  disagree 

10  ■■  9 . 8 . 7 . 6 . 5  4 . 3 . 2 . 1 .  0 

4.  I  was  satisfied  with  the  Project  Definition  process,  because  I  believe  the  process 
enhanced  the  final  product. 


sUAtly  ,  subtly 

agree  **®'dral  disagree 


disagree 


strongly 

disagree 

--  0 


strongly 

disagree 


strongly  sli^tly  ,  sbghtly 

agrS  agree  disagree 

10  --  9 . 8 . 7 . 6 . 5  . 4 . 3 . 2 . 1  • 

5.  Please  provide  any  comments  or  suggestions  you  have  on  hiq)roving  the  Air 
Force’s  Project  Definition  process.  Your  fi-ank  and  honest  input  to  this  study  is 
critical. 


neutral 


slig^itly 


disagree 


strongly 

disagree 


Thank  vnii  for  your  cooperation  in  filling  out  this  questionnaire.  Please  enclose  the 
completed  questionnaire  in  the  envelope  provided  and  mail  it  by  19  Jul  95. 
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Appendix  7 
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Example  Project  DeHnition  Matrix 
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Appendix  8 

Project  Manager  Comments 

The  following  comments  pertaining  to  Project  Definition  were  taken  fi'om  the  Air 
Force  Project  Manager  Surveys: 

The  decision  maker  for  the  entire  project/organization  needs  to  be  identified  at  the 
very  beginning  and  his/her  decisions  followed  without  second  guessing  or  changing 
by  others  later. 

A-E’s  should  consider  contaminated  site  soil  assessment  before  designing  project. 

The  environmental  assessment  should  be  completed  or  be  on  its  way  to  being 
completed. 

Many  etrvironmental  studies  had  to  be  taken  because  Project  Definition  identified 
holes  in  the  requirements. 

Improve  cost  information  at  Project  Definition. 

Project  Definition  process  worked  well  and  minimized  scope  and  programming 
changes  caused  by  force  structure  impacts. 

I  don 't  recognize  any  need  to  improve  the  process;  the  challenge  is  to  adapt  the 
details  of  the  process  to  the  unique  mix  of  personalities,  skills,  and  experience  of  the 
key  team  members  and  the  specific  project  requirements. 

...the  cooling  off period for  reflective  review  was  beneficial  after  the  first  Project 
Definition  presentation. 
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Appendix  9 

Architect-Engineer  Comments 

The  following  comments  pertaining  to  Project  Definition  were  taken  firom  the 
Architect-Engineer  Surveys: 

The  Project  Definition  went  well.  A  key  element  to  its  success  was  the  involvement  of 
the  environmental  department  from  the  beginning.  Had  their  input  not  been  solicited, 
design  changes  would  have  been  the  result. 

The  Project  Definition/on-site  design  charrette  was  instrumental  in  providing  a 
smooth  design  process  without  major  changes,  backtracking,  or  rework.  Its  results 
were  a  superior  product,  responsive  to  the  users  needs,  accomplished  within  schedule 
and  budget.  I  highly  recommend  that  this  process  continue  on  future  MILCON 
projects.  It  works;  ‘‘it  ain 't  broke,  don ’t  fix  it.  ” 

Project  Definition  must  occur  prior  to  or  along  with  establishing  the  project 
construction  budget.  If  the  budget  is  established  prior  to  Project  Definition,  the 
project  is  arbitrarily  constrained. 

A  facility  “design  criteria  ”,  developed  by  the  facility  user  is  also  a  good  method. 
However,  some  users  may  not  have  the  capabilities  to  develop  one. 

Corps  of  Engineers  and  User  Representative  personnel  changes  causes  breakdowns 
and  a  loss  of  knowledge  of  decisions  made  earlier  in  the  project. 

Our  schedule  was  compressed... we  could  not  have  produced  this  amount  of  work  in 
such  a  short  time  unless  all  user  input  was  collected  at  the  beginning  of  the  project. 
The  Project  Manager’s  Guide  to  Project  Definition  is  a  comprehensive,  common  sense 
approach  to  architectural  programming  and  concept  design.  We  support  the 
continued  use  of  this  process. 

A  lessons  learned  conference  at  the  site  would  be  helpful  and  beneficial  to  all 
participants  at  the  completion  of  construction  or  at  90%  completion  of  construction. 

I  like  the  Project  Definition  process,  because  it  very  clearly  spells  out  everything  that 
is  required  for  the  final  product.  The  process  saves  the  A-E  design  team  a  fair 
amount  of  time. 
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Having  the  project  defined  more  clearly  prior  to  beginning  Project  Definition  would 
greatly  improve  the  A-E’s  response  time  during  Project  Definition. 

A  qualified,  knowledgeable  person  should  evaluate  the  “electronic  input  ’’  (base 
sheets)  prior  to  giving  it  to  the  design  team. 

Recommend  a  “configuration  change  board”  be  established,  for  each  project,  to 
review  and  approve  changes  to  scope  of  design,  PA,  CCL,  and  contracting  strategies 
after  final  project  definition  approval.  The  board  should  consist  of  team  members 
familiar  with  the  project  but  not  participating  in  the  day  to  day  design  and  project 
management  effort. 

Recommend,  as  a  part  of  Project  Definition,  the  contracting  strategy  and  bid 
schedule  be  developed  and  approved. 

Having  everyone  involved  with  the  building  together  to  have  an  open  discussion 
about  the  building  program,  organization,  aesthetics,  etc.  was  extremely  productive. 

It  made  a  lot  more  sense  than  a  designer,  with  no  working  knowledge  of  haw  the 
facility  will  be  used,  sitting  alone,  trying  to  make  decisions  about  which  spaces  can  be 
trimmed  or  moved  in  order  to  get  the  building  within  the  programmed  square 
footage. . .  Overall  it  seemed  to  be  a  much  better  way  to  handle  the  project  definition 
and  early  design  process  as  long  as  everyone  who  is  included  is  relevant  and  no  one 
bogs  down  the  meeting  with  their  own  personal  petty  agenda.  There  was  very  little  of 
this  in  our  meetings  but  the  potential  for  this  is  very  high.  Overall — excellent— please 
continue  this  format. 

The  Project  Definition  process  proved  very  effective  in  developing  this  project, 
actually  leading  to  an  Air  Force  Design  Award. 

...Project  Definition  was,  in  fact,  no  more  than  an  exercise  in  schematic  design.  Few, 
if  any,  of  the  other  design-related  agencies  (MAJCOM,  BCE,  Corps  of  Engineers) 
had  any  insight  into  the  process  or  purpose  of  “project  definition  ”. 
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Appendix  10 

Base  Civil  Engineering  Staff  Comments 

The  following  comments  pertaining  to  Project  Definition  were  taken  fi'om  the  Base 
Civil  Engineering  Staff  Surveys: 

Charrette  process  is  good.  Problems  could  arise  if  the  Programmed  Amount  is  too 
low  to  meet  valid  user  requirements.  It  is,  however,  time  well  spent. 

Make  sure  the  base  project  manager  “buys  off’  on  the  Project  Definition. 

The  charrette  process  is  a  great,  efficient  process;  it  was  just  a  little  difficult.  The 
user  and  customer  are  very  pleased. 

Charrette  process  is  time  well  spent.  It  tends  to  define  the  ‘ideal”  facility  early-on. 
Could  cause  some  customer  disappointment  when  the  facility  funding  is  factored  in 
and  some  “nice  to  haves”  have  to  be  cut  from  the  facility. 

On  design-build  projects,  the  project  goals  seem  to  “blur  ”  once  the  design-build/fast- 
track  effort  starts. 

The  “team  in-action  ”  was  evident  even  though  charrettes  were  not  used. 

There  was  simply  no  substitute  to  the  combined  Command,  Wing,  A-E  and  local  user 
team  spending  two  weeks  on  station.  This  produced  not  only  a  Project  Definition  but 
a  Preliminary  submittal  as  well. 

The  facility  users  should  identify  their  organization 's  requirements  prior  to  the  start 
of  Project  Definition. 

Project  Definition  replaced  a  very  much  superior  process  using  a  lengthy  checklist 
(this  should  have  never  been  abandoned)  that  provided  the  information  needed  to 
start  the  designer  on  his  way.  Designers  are  now  tasked  to  provide  the  Project 
Definition  which  is  expensive  and  does  not  necessarily  improve  the  final  product.  A 
good  A-E  should  be  able  to  proceed  (as  they  did  for  years)  with  the  old  checklist  that 
was  done  carefully  by  the  base  C.E.  We  seem  to  think  that  a  change  is  always  for  the 
better. 
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Appendix  11 

Design  Agent  Comments 

The  following  comments  pertaining  to  Project  Defimtion  were  taken  from  Design 
Agent  Surveys: 

The  Project  Definition  process  is  a  good  process,  and  it  is  the  best  to  come  along  in  a 
long  time.  However...  until  the  user  is  willing  to  get  off  his  tail  and  get  involved  in  the 
project  Irom  the  beginning,  the  project  is  doomed  to  a  costly  desigp. 

The  key  to  the  Project  Definition  success  for  this  project  was  the  result  of  an 
innovative  design  approach  by  the  A-E  and  the  determined  coordination  by  the 
AFCEE  PM.  The  PM  was  willing  to  make  on-the-spot  decisions  at  the  charrette. 

This  saved  weeks  (probably  months)  of  time  and  allowed  the  A-E  to  make  some  very 
innovative  design  choices,  saving  the  Air  Force  significant  construction  dollars. 

The  decision  makers  need  to  delegate  the  design  responsibility  to  their  PM’s  and 
honor  their  commitments.  Project  requirements  changed  frequently  due  to  decision 
maker  changes. 

The  use  of  the  charrette  probably  had  no  impact  on  this  project  since  a  key  command 
player  came  in  late  and  changed  requirements. 

There  is  a  misconception  that  charrettes  produce  a  product  at  a  lower  cost  The  A-E 
community  treats  the  charrette  as  an  additional  indirect  project  design  cost  You  pay 
to  design  the  facility  plus  you  pay  an  additional  amount  for  the  charrette.  If  the  Air 
Force  (customer)  could  perform  the  charrette  internally  prior  to  getting  a  designer 
involved,  you  could  probably  save  some  design  money. 

Recommend  that  all  DIs,  RAMPs,  DD  Forms  1391,  other  criteria  and  design  funding 
be  provided  to  the  design  agent  in  accordance  with  the  Military  Construction 
Program  Goals,  i.e.  March  of  FY  minus  2. 

It  is  important  that  input  jrom  BCE/Planning,  Communications,  Security,  &  Fire 
Department  also  be  included  during  Project  Definition,  particularly  to  be  available 
during  the  charrette  process. 

...The  Project  Definition  process  may  be  improved  on  other  projects  if  the  customer 
has  better  thought  out  the  entire  process  and  project  requirements.  Many  times  the 
end-user  is  not  notified  (or  hasn  't  been  communicated)  of  the  importance  of  knowing 

71 


exactly  what  they  ’re  expecting— definite  goal— at  the  project  inception/criteria  review 
conference. 

Adequately  program  project  funding!... There  are  no  excess  funds  to  increase  the 
PA.. .Design  has  proceeded  on  several  projects  with  CWE  above  PA  to  meet  users 
needs. 

Someone  needs  to  explain  the  process  to  the  unit  commanders.  They  don  7  have 
experience  with  the  MCP  process  and  don  7  understand  project  limitations  or 
schedule  requirements. 

Project  Definition  is  fine  when  the  particular  people  are  present.  Many  times  a  few 
critical  people  are  absent  due  to  TDY,  training,  or  poor  meeting  establishment 
date(s). 
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Appendix  12 


Facility  User  Comments 

The  following  comments  pertaining  to  Project  Definition  were  taken  fi'om  the  Facility 
User  Surveys: 

Too  little  realization  that  productivity  is  a  driver— most  of  upper  level  concern  is 
“make  it  fit,  get  it  done  ’’—this  needs  to  change.  Also,  initial  cost  estimates  too  law. 

All  phases  of  the  project  need  to  be  reviewed  to  reduce  “total  time  ”  of  the  project.  I 
watched  two  complete  malls,  two  shopping  centers,  and  a  race  track  constructed  and 
am  still  waiting  for  two  buildings  to  be  finished.  The  process  is  too  long. 

Concentrate  on  the  Corps  of  Engineers. 

The  charrette  serves  a  valuable  service  of  early  team  identification,  customer  buy-in, 
and  functional  accountability — very  worthwhile.  Cost  (programmed  amount)  still 
drives  overall  scope  and  execution. 

It  seemed  as  the  process  progressed,  we  had  less  and  less  say  in  the  decisions  made. 
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Appendix  13 

HRPWC  Comments 

The  following  comments  pertaining  to  Project  Definition  were  taken  fi’om  the 
HRPWC  Surveys: 

Excellent  results. 

Outstanding  process  from  user  standpoint. 

Tremendous  flexibility:  truly  enabled  our  unit  to  tailor  our  facility  to  unique  mission 
requirements.  PM  did  a  fantastic  job. 

Advance  written  notification  of  PD  meeting  with  bullet  comments  of  what  to  have 
prepared  for  the  PD  would  help  “users  ”  come  adequately  prepared. 

The  charrette  process  was  exceedingly  productive  in  providing  a  quality  building 
design.  By  bringing  all  interested  parties  together  in  one  room,  the  required  facility 
elements  were  efficiently  and  effectively  packaged  in  a  timely  manner. 
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